2023 was relatively stable in terms of state-of-the-art models with OpenAI and Llama being all-dominant in the closed and open-weight categories respectively. In 2024 we’re seeing much more competition, including 3 notable entrants this week alone.
Analytics and ML firm Databricks is going “AI-native”, launching a large open-weight model, DBRX, that looks set to become the most powerful freely available. Its slightly above GPT3.5 level but with more efficiency and interestingly trained on more tokens (12 trillion) than GPT-4. Sources suggest a $10m training cost over 2 months on around 3,000 Nvidia H100s, providing a sense of the resources needed for a company to make a meaningful entry into this space.
The Israeli startup AI21 labs announced open-weight Jamba, a hybrid model that combines the typical LLM transformer architecture with a state-space (mamba) approach. This intriguing option should offer better performance over long inputs, and combine the reasoning power of the transformer with the supreme memory efficiency of the SSM.
On Thursday Musk’s xAI announced Grok 1.5, taking their X.com embedded model (not yet available in the UK, although accessible to Premium+ subscribers using a VPN), with GPT-4 class performance and a long 128k token input window. Musk also posted that “Grok 2 should exceed current AI on all metrics. In training now.”
Takeaways: The GPT-4 class level is getting increasingly crowded, although Anthropic’s Claude 3 family is inching ahead. Check out the community powered LMSYS model leader board showing Opus in top spot, Haiku the super low-cost and fast model entering the top ten, and Google’s Gemini models also performing well. For further insight into model performance and cost we also recommend Artificial Analysis. All eyes remain on what will be a defining AI summer, with GPT-5 and Llama3 setting out the next level of capability.