GPT-4 (the model in ChatGPT Plus, the paid service, and behind many advanced AI tools) has spent the last year or so as the undisputed (LLM) AI champion of the world. But in the space of a few weeks the ring has become much more crowded. ExoBrain have concluded that Claude 3 Opus is the new heavyweight, particularly in a qualitative sense due to its ‘meta-cognitive’ abilities. But there are challengers to OpenAI in every class. We’ve put together a comparison, based on industry data and our own assessments. Below we plot AIs against capability (scores on standard tests and input window size etc.) and cost.
To provide a sense of the range of costs, based on the number of words or images going in and out, the most expensive a Claude 3 Opus powered assistant, would set you back 60p to read 10 pages of A4 business content and generate about the same in analysis. Whereas, the French contender’s Mistral 7B, a fly-weight model in comparison, would cost less than a penny or indeed 180x less. Clearly such a model’s capabilities are also much more limited, it can do only basic reasoning on a max of 16 pages at once, but will do that 3x faster.
We’ve also pencilled in some of the new and upcoming models. Inflection’s Pi got a big upgrade on Thursday and is now GPT-4 class and 40% of the size (and likely) cost, and Google’s next Gemini models are nearing wider availability.
We would highlight 3 clusters of AIs:
- State of the art, expensive, deep and untapped power: Claude 3 Opus and the yet to be released Google Gemini Ultra 1.5
- Excellent general purpose all-rounders: GPT-4, Claude 3 Sonnet and Pi (although its conversational style is not suited to all)
- Fast and cost-efficient automation workhorses: Gemini Pro, Chat GPT3.5 and Mixtral 8x7B
- Click here for a hi-res version
