Week 50 news

This week we look at:

• Google’s Gemini 2.0, multimodal capabilities, and world models.
• The release of Devin, an autonomous AI developer reshaping workflows for software engineering.
• AI’s growing and sometimes controversial role in healthcare.

Gemini through the looking glass

Google’s Gemini 2.0 debuted this week, almost exactly one year on from the first Gemini release. The initial ‘2.0 Flash’ model, an efficient workhorse version, introduces true real-time video ingestion and native multimodal capabilities, and performance that punches well above its weight. This suggests a lot is to come from the more powerful siblings. The Gemini 2.0 comms barrage also included a raft of new products, projects and ideas, many of them around the notion of seamless switching between text, visuals and audio, and agents… for world understanding (Astra), coding (Jules), and browser tasks (Mariner). Not to be outdone, OpenAI simultaneously announced screen-sharing and live video features for ChatGPT’s Advanced Voice Mode, enhancing its ability to assist users in real-world activities.

At the core of Gemini 2.0 is its ability to understand and generate text, images, audio, and video interchangeably. Google believes ‘multimodality’ is one of the keys to creating AI systems that can work truly autonomously. The Multimodal Live API made available this week on the Google AI Studio allows developers to build systems that interpret visual data in real time, engage in dialogue, and execute tasks. A fluid voice chat that involves the AI being able to look at your screen or see what you are seeing and provide insights is impressive. OpenAI have likely been holding the roll-out of their live frame-by-frame video (demonstrated back in May) for just this moment. They duly released as the social media buzz around the Gemini 2.0 was developing, with people posting fun examples from completing maths homework to suggesting cocktails from observing a shelf of bottles. OpenAI’s audio and video experiences at this stage feel more polished, and whilst it’s taken many months, the rich interactions promised earlier in the year are now here.

But where Gemini 2.0 is different is the move from text fully interchangeable modalities, and what Demis Hassabis calls ‘world models’. Google is moving beyond internet training data, and the structured, pre-labelled visual data towards something more akin to human learning. Oriol Vinyals, co-tech lead of the Gemini project suggests that while current models excel at connecting concepts present with textual descriptions, they haven’t yet cracked developing true world understanding. Unlike a child who can watch objects fall and gradually build an intuitive understanding of gravity, many current AIs still rely heavily on human-provided text descriptions to make sense of visual information. But Google is determined to overcome that limitation and exploit the untapped potential of video understanding as a vast repository of knowledge about physics, causality, and natural laws – knowledge that exists independent of intrinsically limited human annotation. While models can tell you what’s happening in a video, they can’t yet extract fundamental principles from pure observation, at least for now. It’s the difference between describing a falling apple and deriving Newton’s laws of motion. Imagine a training regime that could extract fine grained insights from high-definition video, this is a world away from a brief textual explanation of a static image.

The introduction of OpenAI’s Sora may appear tangential to Gemini 2.0’s multimodal capabilities, but it actually sits at the heart of the same fundamental challenge – developing world models. While Sora’s immediate application is video generation, OpenAI is pursuing a similar goal to Google: teaching AI to understand physical laws and causality. What makes Sora particularly relevant is how it appears to have developed an intuitive grasp of physics, motion, and object persistence – not just through labelled data, but through learning to predict how scenes naturally unfold.

Takeaways: Google’s Gemini 2.0 release represents a huge bet on true multimodal AI rather than just connecting different modes through text, and OpenAI with Sora are making the same bet albeit via a slightly differing track. Both firms believe that to create robust, autonomous agents they must have a better understanding of the world. Humans can resolve issues with the tools and technology around us because we have spent years problem solving in the physical environment, and agents will need to do the same. LLMs can mimic human communication, new reasoning models are starting to solve complex scientific problems, will world models truly and independently understand the world? 2025 may be the year we find out.

Devin joins the team

Devin, the autonomous AI developer from Cognition Labs that we covered back in March, is now finally available, and this week engineering teams all around the world, including ExoBrain’s, have been on-boarding the shape of things to come. So far, we’ve been impressed; Devin is a capable and thorough team member, and most of all, is very willing to learn. With a clever memory feature, plus configurable playbooks, we can see Devin becoming ever more familiar with our projects, and able to handle increasingly complex tasks. We’ve been integrating Devin into our workflow via Cursor’s IDE extension, which keeps us updated on its activity, and Slack, where it can manage multiple conversations at once. Despite its strengths, Devin still benefits from super clear guidance and well-scoped tasks to ensure it stays on track. Its API opens up even more potential, such as programmatically launching instances to execute security patches or automate specific workflows.

With models improving and costs reducing, these autonomous colleagues can only improve; this is a glimpse into the future of many technical jobs. Devin is also part of a bigger picture. There is a growing ecosystem of AI tools shaping a new way to build software. With tools like Devin, Gemini Deep Research, Pinokio, o1 Pro Mode, Bolt, and Cursor working in tandem, it’s possible to employ an AI-enabled assembly line where ideas flow rapidly from strategy to deployment:

• The newly released Gemini Deep Research (which uses a web browsing agent to source tens of documents and pages before writing a report in Google Docs), plus Perplexity’s AI search start the process with their ability to support market research and concept exploration.
• OpenAI’s o1 Pro Mode can significantly extend these early ideas with strategic insights and detailed architectural planning and design.
• Pinokio, meanwhile, is a useful piece of the puzzle, offering a quick way to find, install, and test the latest AI tools and open-source projects.
• Bolt.new accelerates prototyping in the browser, leveraging Claude’s coding capabilities and web containers, ideas can be realised in a fraction of the typical time to create a testable prototype.
• Cursor is emerging as the AI development powerhouse (with Windsurf hot on its heels). It facilitates accelerated coding with its integrated multi-model chat, slick in-line AI editing, and composer agents that can create and edit files autonomously. It also seamlessly pulls in external documentation, design documents, and coding standards to ensure the AI development is informed and aligned.
• Vercel’s v0 is the AI-powered tool changing how developers create user interfaces. It offers a chat-based approach to generate production-ready UI components using modern web technologies like Next.js and React.

And finally, Devin can handle the heavy or time-consuming work once your product is up and running, testing new features, making repetitive changes, analysing bugs, and many of the hard yards needed to keep software up to date and robust over the longer term.

Takeaways: Devin and a range of other AI tools are reshaping how software is built. When combined thoughtfully, they can speed up development from initial research through to deployment and maintenance. While software engineering is an early testing ground, the lessons learned here will help guide AI adoption across many other technical fields.