How do you measure whether an AI is actually good at predicting the future? The CTO of startup Obside emailed me recently with a fascinating real-world benchmark.
Instead of giving AI models another standardized test, Obside had ChatGPT, Gemini, Claude, Grok, Mistral, DeepSeek, and Kimi bet on World Cup matches using live Polymarket odds.
An hour before kickoff, each model goes into agent mode, researches the teams, injuries, and other public information, then decides how much of its virtual $10,000 bankroll to wager.
When I checked after the semifinals on Thursday, French open-source darling Mistral led the field, followed by OpenAI's GPT 5.5, and DeepSeek's V4. Claude Opus 4.8, meanwhile, sat firmly at the bottom, the only model in the red. Maybe Anthropic's AI is simply too ethical to be a gambler?
Check out the current standings here.
This fun exercise measures something many AI benchmarks can't: judgment under uncertainty. That's a theme I've explored before. Last year, I wrote about ChatGPT entering a secret forecasting tournament run by economists, where it performed no better than the average human contestant.
Betting on soccer isn't the same thing, but it's another clever way to test whether AI can turn online information into profitable predictions when nobody yet knows the answer.
Sign up for BI's Tech Memo newsletter here. Reach out to me via email at [email protected].
Read next
Alistair Barr is the author of Business Insider's Tech Memo newsletter. Sign up here. Before that, he was BI's Global Tech Editor and the Big Tech team leader at Bloomberg, following a reporting career at The Wall Street Journal, USA Today, Reuters, and MarketWatch. Alistair won a Gerald Loeb Award in 2007 for coverage of short selling and was a finalist in 2013 for scoops on the Facebook IPO. More recently, he won a 2024 San Francisco Press Club award for commentary. Got a tip? Reach out using the secure messaging app Signal (+1 415-341-4927) or via email on [email protected].ExpertiseAlistair oversees all things Big Tech, along with startups and venture capital. He writes analysis and columns about topics including generative AI, large language models, cloud computing, semiconductors, online search, e-commerce, EVs, robotics, and autonomous vehicles.Popular StoriesArtificial Intelligence:It's getting harder to make big leaps at the frontier of AIOpenAI's AI-adjusted earnings numbers have echoes of Groupon and WeWorkDeath by LLM: Stack Overflow's decline, and its plan to survive, shows the future of free online data in an AI worldCloud computing:Amazon dominated the first cloud era. The AI boom has kicked off Cloud 2.0, and the company doesn't have a head start this time.In cloud, there's AI (which is hot) and everything else (which is not)Chips:Why Intel is still so important: Real countries have fabsApple's made-in-the-USA chips signal a turnaround for the US's big semiconductor betEVs and Tesla:Tesla's AI supercomputer has a Silicon Valley town rushing to meet surging electricity demandTesla's Cybertruck is outselling almost every other EV in the USOnline Search:Google is losing its status as a verbA simple way to fix search: Bright pink ads
Follow Following
Every time publishes a story, you’ll get an alert straight to your inbox!
Look out for an alert in your inbox the next time publishes a story!
Every time a new story is published, you’ll get an alert straight to your inbox!
Look out for an alert in your inbox the next time a new story is published!
By clicking “Sign up”, you agree to receive emails from Business Insider. In addition, you accept Insider’s Terms of Service and Privacy Policy.
