K2-Horizon: A Frontier Fleet of Open Source Language Models

Abstract

We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks. On the modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating, a more flexible, theoretically motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2× without a separate draft model. Our release follows the LLM360 definition of open source for transparency and reproducibility: the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself, we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/.

Type
Publication
Technical report

This is the technical report for K2-Horizon, a fleet of six open-source language models from MBZUAI’s Institute of Foundation Models (IFM).

My contribution, stemming from my internship in the summer of 2026, is Section 5.12.5: a multi-agent research community simulation.

It simulates a persistent, real-time world in which researcher agents (each with its own persona, discipline, and skillset) work on a shared pool of research tasks, both collaborating and competing. They coordinate over direct messages, group chats, file sharing, and shared git repositories; they buy and sell work and services on a marketplace for an in-game currency, and interact with a wider audience of hundreds of simulated public accounts corresponding to journalists, students, and aggregators on an X-like social network.

They submit papers to a conference with a deterministic verifier and an LLM referee, compete on a leaderboard for hard open-ended problems, strive to gain the most influence on the social network, or simply amass the most currency by performing tasks for others. A second, collaborative configuration is targeted towards the hardest tasks: it removes the competition and is designed to study emergent collaborative patterns of many agents working toward a shared goal.

As the simulation is non-blocking (the world moves on as the agents think), it results in very dynamic trajectories where, for example, agents have to pivot their research having been “scooped” by another, forming alliances, and much more. Such trajectories thus provide dynamic training data that cannot be obtained from single-agent trajectories and were used to train the K2-Horizon family of models.

Andrej Leban
Andrej Leban
Ph.D. Student