Krafton Unveils Language Model-Powered Game AI Evaluation Tool 'Orak'

Krafton has opened new horizons in game AI research by unveiling 'Orak,' an innovative benchmark designed to evaluate the game performance capabilities of AI agents based on Large Language Models (LLMs). Inspired by the familiar Korean word 'Orak' (meaning entertainment/amusement), Orak is designed to quantitatively analyze an AI's situational awareness, judgment, and action-making processes across six core game genres: action, adventure, RPG, simulation, strategy, and puzzle. This enables a deeper assessment of AI intelligence beyond mere gameplay. Orak is based on the AI design experience accumulated through CPC, which was jointly developed by Krafton and NVIDIA. In particular, the core technology, MCP (Main Control Protocol), plays a decisive role in making LLM agents' game play a reality. MCP acts as a bridge, converting complex game information into text-based information that language models can understand, and translating LLM responses into executable in-game actions. Through this, LLMs are evaluated on their ability to interpret situations and make decisions independently within complex game contexts, much like a real player: perceiving game states through text or images, identifying obstacle locations, making appropriate judgments like movement or jumping, and then selecting actions. Krafton plans to expand its research through Orak into various industrial sectors beyond the game industry in the future, and will also provide a dataset for LLM fine-tuning, enabling researchers to optimize pre-trained LLMs for actual game environments. Lee Kang-wook, Head of Deep Learning at Krafton, stated, “Orak is a game-specialized LLM benchmark that embodies Krafton’s pioneering research and know-how,” adding, “We are also planning competitions in the future to challenge LLM agent design capabilities based on this,” which is garnering anticipation. Krafton will continue to develop LLM technology optimized for the gaming sector.