Executive Summary
Google DeepMind has introduced a new framework based on cognitive science to systematically measure the progress of Artificial General Intelligence (AGI). Detailed in their paper, "Measuring Progress Toward AGI: A Cognitive Taxonomy," the framework defines 10 key cognitive abilities and a protocol for benchmarking AI systems against human performance. To put this theory into practice, the company has launched a Kaggle hackathon with a $200,000 prize pool, inviting the research community to build the necessary evaluations.
Key Takeaways
* New Framework: The announcement introduces a cognitive taxonomy to scientifically evaluate the capabilities of AI systems on the path to AGI.
* 10 Cognitive Abilities: The framework identifies 10 key abilities as crucial for general intelligence: Perception, Generation, Attention, Learning, Memory, Reasoning, Metacognition, Executive functions, Problem solving, and Social cognition.
* Evaluation Protocol: A three-stage protocol is proposed: evaluate AI on a broad suite of cognitive tasks, collect human performance baselines for the same tasks, and map the AI's performance relative to the human distribution.
* Kaggle Hackathon: A competition titled “Measuring progress toward AGI: Cognitive abilities” has been launched to crowdsource the creation of evaluations.
* Hackathon Details: The hackathon offers a $200,000 prize pool and focuses on building evaluations for five key areas: learning, metacognition, attention, executive functions, and social cognition. Submissions are open from March 17 to April 16.
Strategic Importance
This initiative positions Google DeepMind as a leader in defining the scientific measurement of AGI, moving the industry discourse from abstract concepts to a structured, empirical evaluation framework. By involving the community, they accelerate the development of standardized benchmarks while solidifying their research leadership.