Meta has shaken up the AI race with a release of pre-trained models that rely on a new multi-token prediction approach with the potential to transform how large language models (LLMs) are designed and deployed.

The new method breaks through the cemented AI training mold, which typically trains LLMs to predict the next word in a sequence based on previous words.

Meta’s novel model instead is trained to provide multiple words at the same time, which aims to enhance performance and reduce training times.

First revealed in a research paper in April, the new model is meant to be especially efficient in generating code, and is useful for larger model sizes.

The multi-token prediction approach is meant to decrease the swath of data required to train LLMs, as well as lessen its negative environmental impact.

By using the same amount of input to produce multiple outputs at the same time, it can increase efficiency and could potentially remove some of the hallmark issues surrounding AI training.

Recommended reading

• UK Putting £8.5M Into Research That Helps Protect Society From AI Risks
• Scots Unis to Take Part in £3.5M Project to Reduce AI Harm
• Navigating the Ethical Implications of AI ‘Deadbots’

As large language models expand, they require more and more data to train on, which requires data centres to store and transfer, requring mass amounts of energy and water. Concern around the carbon emissions and environmental impact of AI has grown, especially since Google’s annual environmental impact report cited AI as the main obstacle to it reaching its net zero goals.

To help develop the multi-token prediction, Meta has released the pre-training models for code completion, under a research only and non-commercial license.