A novel methodology devised by the highly anticipated Sakana AI, a startup based in Tokyo, Japan, generates generative models automatically. The methodology, known as Evolutionary Model Merge, draws inspiration from the natural selection process by integrating components from pre-existing models in order to generate more proficient models.
Sakana AI, which was co-founded by eminent AI researchers including former Google employees David Ha and “Attention Is All You Need” co-author Llion Jones (the paper that ushered in the current generative AI era), made its initial public appearance in August 2023.
Sakana’s novel Evolutionary Model Merge methodology may empower organizations and developers to generate and uncover novel models in a cost-efficient manner, obviating the necessity to invest substantial resources in training and refining their own models.
Sakana has made available a vision-language model (VLM) and a large language model (LLM) that were jointly developed using Evolutionary Model Merge.
Model Integration
Training generative models is a complex and costly endeavor that beyond the financial means of the majority of organizations. However, since the introduction of open models like Llama 2 and Mistral, developers have devised inventive, low-cost methods to enhance them.
“Model merging,” in which various components of two or more pre-trained models are combined to form a new model, is one such technique. When executed accurately, the merged model has the potential to acquire the merits and functionalities of the individual parent models.
Interestingly, additional training is not required for merged models, which renders them extremely cost-effective. Indeed, numerous Open LLM leaderboard-performing models are merged iterations of well-liked base models.
Sakana AI’s researchers write on the company’s blog, “What we are witnessing is a large community of hackers, enthusiasts, and artists going about their own ways of developing new foundation models by fine-tuning existing models on specialized datasets or merging existing models together.”
Hugging Face provides an extensive collection of over 500,000 models, and model merging presents organizations, researchers, and developers with limitless opportunities to investigate and generate new models at a nominal expense. Nevertheless, model merging is predominantly dependent on domain expertise and intuition.
Merge Of Evolutionary Models
The objective of the new method developed by Sakana AI is to provide a more methodical method for identifying efficient model merges.
Researchers at Sakana AI write, “We believe evolutionary algorithms, inspired by natural selection, can unlock more effective merging solutions.”
Evolutionary algorithms are optimization techniques that operate on populations and draw inspiration from the processes of biological evolution. Candidate solutions are generated iteratively by combining elements from the extant population and employing a fitness function to determine the most optimal solutions. Evolutionary algorithms are capable of investigating an extensive realm of potentialities, uncovering unique and illogical amalgamations that conventional approaches and human intuition may fail to perceive.
David Ha, founder of Sakana AI, told VentureBeat, “The ability to evolve new models with new emergent capabilities from a vast array of existing, diverse models with various capabilities has significant implications.” “In light of the escalating expenses and resource demands associated with training foundation models, large institutions or governments may contemplate adopting the more cost-effective evolutionary approach by utilizing the abundant assortment of foundation models in the thriving open-source ecosystem.” This would enable them to rapidly develop proof-of-concept prototype models prior to allocating substantial capital or national resources towards the development of entirely custom models from the ground up, assuming such endeavors are even necessary.
The Evolutionary Model Merge, a general method developed by Sakana AI, employs evolutionary techniques to determine the optimal methods for combining distinct models. Evolutionary Model Merge generates and assesses new architectures by autonomously combining the weights and layers of existing models, as opposed to relying on human intuition.
Regarding Sakana AI:
“Our approach leverages the extensive collective intelligence of pre-existing open models to generate new foundation models with user-specified desired capabilities in an automated fashion,” Sakana writes on his blog.
Active Evolutionary Merging
Having observed the remarkable progress made in manually integrated models, the researchers were intrigued by the potential of an evolutionary algorithm to discover novel approaches for combining the vast collection of open-source foundation models.
Evolutionary Model Merging, they discovered, uncovered non-trivial methods for merging models from drastically different domains, such as non-English language and mathematics or vision.
“We initially evaluated our method’s ability to automatically generate a Japanese Vision-Language Model (VLM) and a Japanese Large Language Model (LLM) capable of mathematical reasoning in order to test our approach,” the researchers write.
Without explicit optimization, the resulting models attained state-of-the-art performance on a number of LLM and vision benchmarks. The evolutionary algorithm was employed to integrate the Japanese LLM Shisa-Gamma and the math-specific LLMs WizardMath and Abel into a single LLM.
Their 7-billion-parameter Japanese math LLM, EvoLLM-JP, outperformed some state-of-the-art 70-billion-parameter Japanese LLMs and attained high performance on a number of Japanese LLM benchmarks.
The researchers write, “We believe our experimental Japanese Math LLM is adequate to serve as a general-purpose Japanese LLM.”
They utilized LLaVa-1.6-Mistral-7B, a well-known open-source VLM, and Shisa-Gamma 7B for the Japanese VLM. The resultant model, EvoVLM-JP, not only surpassed LLaVa-1.6-Mistral-7B but also JSVLM, an established Japanese VLM, in terms of performance. Both models were made available on Hugging Face and GitHub.
Additionally, the group is advancing in its utilization of evolutionary model merge techniques for image-generation diffusion models. They are developing a new version of Stable Diffusion XL that generates images extremely quickly and with high-quality responses to Japanese prompts.
“We obtained the EvoSDXL-JP results just days prior to release, so we have not yet published or written up a comprehensive writeup for that model.” “With any luck, that one will be available within the next one to two months,” Ha stated.
Sakana AI’s Objective
Sakana AI was established by Ha, a former Google Brain researcher and director of research at Stability AI, in collaboration with Llion Jones, a co-author of the groundbreaking 2017 research paper that introduced the Transformer architecture utilized in generative models.
Sakana AI focuses on developing novel foundation models by implementing concepts inspired by nature, such as collective intelligence and evolution.
The researchers wrote, “Instead of a solitary, enormous, all-knowing AI system that demands enormous amounts of energy for training, operation, and maintenance, the future of AI will consist of a vast collection of small AI systems—each with its own niche and specialty—that interact with one another, with more recent AI systems designed to fill specific niches.”