Meta has released four open-source pre-trained AI models that leverage a new approach to machine learning, known as “multi-token prediction,” which could help inform how large language models are developed.
According to the company, this new technique differs from the traditional method of training large language models by predicting the next word in a sequence, as it predicts multiple future words at once, which promises improved performance and reduced training times.
As AI models grow in size and complexity, the increased demand for computational power has raised concerns about cost and environmental impact, the company explained. The “multi-token prediction” method may provide a way to mitigate this trend, making advanced AI accessible and sustainable.
“By predicting multiple tokens simultaneously, these models may develop a fine-grained understanding of the architecture and context of language,” she added.
The company noted that the new models could lead to improvements in tasks ranging from code generation to creative writing, potentially bridging the gap between artificial intelligence and human-level language understanding.