001: Beware Danger Beyond the Goldilocks Zone
Welcome to Predictions: A New Forecasting Newsletter by (formerly) Global Guessing and Crowd Money
Welcome, everyone!
For many of you, you probably remember us (Andrew and Clay) from our days with Global Guessing and Crowd Money—two newsletters and podcasts about geopolitical forecasting and prediction markets respectively.
Since then, a lot has happened. Stories for another day. Today we want to introduce you to something new and also not: our newsletter Predictions.
Predictions is a weekly newsletter by us (now GeoVane, you may know us from YouTube). In this newsletter, you can expect:
News in Numbers: The biggest (or sometimes, not biggest) news stories of the week, from forecasters and prediction markets, selected by us.
Opinion: Andrew and Clay offer their opinions, theories, insights, and discussions about the field of quantified forecasting, its application, and the domain of predictions markets.
Global Guessing (optional): Like our original namesake, we apply the practice of quantified forecasting to the domain of geopolitics. Find the top-level takeaways here, and get full access to our live prediction through our Patreon.
Base Rate News: Industry news about forecasting and prediction markets, ala Nuño Sempere’s now retired Forecasting newsletter.
There’s more to this newsletter than just this, with more optional sections appearing. Some of your favorite brands from the past might even make a comeback—but you’ll have to wait for the next issues to see what they will be.
And of course, the podcast is coming back.
❤️ Don’t forget to like the newsletter. It’s like a tip, but free!
💬 Agree or disagree with something you read? Have a suggestion? Leave a comment!
🔁 If you loved the newsletter, please consider restacking it!
📧 Please share Predictions with family and friends! We even set up referral rewards through Substack 🙂
News in Numbers
Six forecasts for your news. Numbers were recorded on August 17, 2023, with series entries recorded either weekly (the first three forecasts) or monthly (the last three). Some forecasts are included because of their significant c hanges in light of news, while others because they have in fact stayed frozen.
Fed's Bostic says U.S. interest rates are high enough (Reuters)
Niger puts military on ‘maximum alert’ over ECOWAS attack fears (Al Jazeera)
Getting Israeli-Saudi Rapprochement Right (Foreign Affairs)
U.S. intelligence says Ukraine will fail to meet offensive’s key goal (Washington Post)
The US and Iran look for de-escalation (Financial Times)
641 years behind bars? No, but Trump’s risk of prison is real. (Politico)
Opinion
If you are reading this newsletter, you probably have a certain amount of intellectual buy-in on the concept of quantified forecasting. However, much of the social sciences remain skeptical, to say the least, about the practice.
Take the domain of international relations, where Philip Tetlock got his start and serves as the basis for much of the quantified forecasting research. Many scholars have raised objections about whether or not we can prediction international relations which can be best captured by Robert Jervis’ System Effects published in 1997.
System Effects
The essence of the book (which should probably get its own dedicated post at some point) is that the realm of international politics deals with a system where its elements are interconnected, such that a change in one part of the system makes changes in other parts of it, and that the system contains properties and behaviors which are different than those of its part (in other words, that the total is greater than the sum of its parts).
As a result of systems effects, in the international system (and any complex, interconnected system for that matter) we have:
Delayed and indirect outcomes
Emergent characteristics — Relationship between elements are based on relationships to other elements
Non-integrable function — you cannot understand the whole thru its parts
Unintended outcomes
Nonlinearities — unexpected breaks from the past given history
Feedback loops
Regulation being difficult
Does Jervis believe these system effects doom prediction? Not entirely, especially in his 1997 book. Although by his 2012 revisit, Jervis takes a slightly more negative tone, writing:
...My approach has an ambiguous stance towards prediction” and that “being realistic about the limits of our ability to know how we can reach desired ends can make us freer to act on our ideals. When it is not possible to see around the bend, to use Jones-Rooy and Page’s phrase, perhaps it is better not to try.
What is the response by the forecasting space to these claims? In the same journal edition as Jervis’ 2012 update, Philip Tetlock along with Horowitz and Hermann argue that:
System effects do not preclude pockets of predictability
Understanding system effects can helpful to expand those pockets
We should focus on questions that are in the “Goldilocks zone” of difficulty (between 10-90% ex-ante)
System Shock
Last month, the third point seemingly became irrelevant as the Forecasting Research Institute published some early results from their long-run forecasting tournament titled Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament.
Per the paper’s abstract, “the Existential Risk Persuasion Tournament (XPT) aimed to produce high-quality forecasts of the risks facing humanity over the next century by incentivizing thoughtful forecasts, explanations, persuasion, and updating from 169 forecasters over a multi-stage tournament.”
This new research is relevant for three main reasons:
The risk areas forecasted in the tournament (nuclear weapons, artificial intelligence, climate change, biorisks) are all topics of significant public interest, and any insights into these risks is noteworthy.
The tournament combined the mental might of both “superforecasters” and “experts,” giving us a unique look into how these two groups of forecasters compare with respect to accuracy.
This tournament is one of the first attempts at applying the short-range forecasting methodologies pioneered by Philip Tetlock to long-range forecasting questions.
Before reviewing the results from the report, there are two interesting choices made by the research team in the experimental setup that are worth mentioning.
First, the report states that 42% of the experts included in the tournament were members of the Effective Altruism community (defined as having attended an EA meetup).
Second, as long-range forecasts naturally will not resolve for decades, the tournament implemented something called intersubjective forecasts in order to measure long-range forecast accuracy, defined as “predictions of the views of other participants.”
Now to the results! If you couldn’t tell from the title of this section, we were not very impressed, or optimistic, about the early findings from the tournament.
The report states that its purpose is to document “variation in probabilistic beliefs and explanatory rationales on high-stakes issues,” but unfortunately it provides very little conclusive evidence of the accuracy of those beliefs.
Most of the findings from the tournament felt administrative, i.e. “facilitating productive adversarial collaborations” or how to “retain the talent of busy professionals in a demanding multi-month marathon.”
The report seems to put the cart before the horse, stating a goal in its Next Steps section to “make these forecasts more relevant to policymakers.” Conclusive evidence on accuracy is a prerequisite to providing policy recommendations based on long-term forecasts.
Initial thoughts
There is much to say and still think about when it comes to long-term forecasting and this report, which we plan to do over the next month as we get ready to give a talk on The Pitfalls and Promises of Long-Term Forecasting at the Manifest 2023 Conference.
The accuracy of human forecasting has long been a contested topic within social sciences such as international relations. That has begun to change as research from IARPA, Philip Tetlock, and others have demonstrated the viability of short-term forecasting, demonstrated in the books Expert Political Judgment and Superforecasting.
Today academics are continuing to push the boundaries of human forecasting research, and a new area of research has begun to gain traction: long-term forecasting. In this talk, Clay and Andrew from GeoVane will explore questions of whether or not humans truly have the tools, cognitive processes, and even capability to make accurate, repeatable, long-term predictions. Despite the clear benefits it would deliver if feasible, we predict that the answer will be no, and that the risks of wasted intellectual capital warrant serious discussion.
So with that being said, these are some of our initial thoughts which will animate our thinking moving forward. The main one being that these findings in conjunction with past readings have naturally led us to consider the merits of long-range forecasting writ large—finding ourselves increasingly aligned with the conclusions in Karl Popper’s Poverty of Historicism.
Ultimately, due to the early results of the tournament, it feels like calling long-range forecasts, “forecasts,” may itself be a misnomer. The long-range forecasts provided by the tournament participants read more like opinion polls, measures of belief, rather than prescriptive predictions about future outcomes.
Short-range forecasting already involves compounding conditional outcomes to generate a probability. This exercise comes with the risk of any individual conditional outcome changing, thereby affecting the final probability. Long-range forecasting increases the number of these conditional outcomes exponentially, to the point where it feels futile to even attempt to control those innumerable variables.
But this assumes that approaching long-range forecasting is even similar to approaching short-range forecasting. And whether or not that is correct is not clear either. In short-range forecasting, it is common to begin with a base rate – the frequency with which a similar event has taken place in the past. For long-range forecasting on topics like existential risks, oftentimes base rates do not exist. How do you approach a forecast where the event in question has never occurred before?
It seems that to get around this issue, the long-range forecasting tournament used intersubjective forecasts as a proxy for long-range forecasting accuracy. And while the research team behind the event views this methodology as adequate, we are not entirely convinced.
Now as we mentioned, the results in this report are early and as the report states, results will continue to filter in over the next few decades. If good research on this form of prognostication can only happen over decades, then even if these forecasts prove to be insightful, the logistics behind progressing the research become impractical.
Instead of trying to create prescriptive long-term forecasts, we should use belief measurements like those provided by this report as data points with which to create frameworks to forecast from. These long-range forecasts (maybe better termed “predictions”) can provide high-level context for short-range forecasts which are evidently more accurate and more importantly, more actionable.
If we want forecasts and forecasting as a field to be more greatly embraced by the policy community, we must focus on the most accurate, actionable areas of our field. And those are short-range forecasts. Tournaments like these, with a plethora of rules and intersubjective forecasting assignments, will lead to burnout for forecasters, and also create a selection bias from some of the minds most relevant to these existential risk discussions.
Global Guessing
In light of the recent coup in Gabon, marking the 4th attempted one in Africa this year, we made our first Patreon-exclusive live prediction on whether there would be another one this year within Francophone-Africa. So if you want to understand our process and how we made this forecast, please subscribe to our Patreon—or if you just want to support us making more content!
Our conclusions, however, were that we forecast a:
67% chance there will be a military coup in another francophone country in Africa before the End of 2023.
And that we are watching:
ECOWAS
Regional winter elections
Rwanda and Cameroon (given their recent military purges)
France’s on-going response
Central Africa Republic, Burundi, and Chad
You can find the question on Manifold Markets:
Base Rate News
Fumbling the Crystal Ball Policymakers Can’t Afford to Spurn the Science of Prediction | Foreign Affairs | 12.16.22
🥩 The Meat: Philip Tetlock and J. Peter Scoblic followed up to their 2020 article, offering a renewed case for why policymakers should adopt quantified forecasting, while offering reasons for the lack of its adoption thus far.
🥔 The Potatoes: Although heavily invested by the intelligence communities, quantified forecasting and prediction markets have received little if any substantive penetration into the policymaking space. Some of the issues raised by the authors, such as the limits of forecastable questions and the different stakes between policymakers and financial traders, are harder than others to overcome.
The Forecasting Research Institute launches | 12.13.22
🥩 Philip Tetlock has launched a forecasting think tank aimed at advancing the current forecasting literature and making forecasts more actionable for organizations and policymakers.
🥔 While any significant allocation of resources towards furthering forecasting is exciting, as we covered in our opinion section, the results thus far have been underwhelming. The group is constituted of some brilliant minds, however, and we’re optimistic that they can produce some watershed research.
Insights into the accuracy of social scientists’ forecasts of societal change | Nature | 02.09.23
🥩 Results from a recent research paper published by The Forecasting Collaborative found that, in forecasting tournaments testing the accuracy of predictions of societal change, “social scientists’ forecasts were on average no more accurate than those of simple statistical models…or the aggregate forecasts of a sample from the general public”
🥔 We already knew that subject matter experts do not necessarily make the best forecasters, especially those whose domains are dominated by theory. Though these results do seem to provide more support for the veracity of crowd forecasts found on platforms like Metaculus.
Nate Silver, FiveThirtyEight Founder, Expects to Depart ABC News Amid Layoffs | NYTimes | 04.25.23
🥩 Nate Silver, founder of FiveThirtyEight, was ousted at ABC News after layoffs across the media and technology industry hit Disney’s news division.
🥔 To be honest, we looked to FiveThirtyEight as a northern star for how to do data journalism when we began our forecasting journeys in 2020. It’s sad to see the platform struggling, but it is also clear that doing news profitably is very hard.
The Future of Futures: On Kalshi and Prediction Markets | Los Angeles Review of Books | 07.09.23
🥩 Addis Goldman and Max Hancock discuss the ethical dimensions of prediction markets and the over-quantification of uncertainty.
🥔 Although proponents of prediction markets, we have often discussed the ethical limits of them. This article raised my interesting questions and discussions between us which we don’t want to bias you towards before hand. Please let us know your thoughts in either our comments or theirs!
A resurgent online betting market is boosted by crypto and current events | NBC News | 07.10.23
🥩 Polymarket came under fire this summer for allowing users to place bets on whether the Titanic Submarine would be found before June 23.
🥔 With a non-zero percentage of new users betting on the submarine question stuck around and mainstream news organizations have turned an eye towards the space, another major news event where prediction markets enter the mainstream—when will the back break?
A new organization Optic Forecasting launches | 04.22.23
🥩 A new organization running war games / Model United Nations-esque games but based on forecasting instead.
🥔 Really cool idea which we have talked about privately before. Excited to partake in an event some time and report back the experience. If you’ve been, how was it?
Base Rate Times launches | Mid-2023
🥩 A website covering major news stories via graphical forecast aggregation.
🥔 Similar to our work prior to and immediately following the Russian invasion of Ukraine, as well as the news in numbers section in this newsletter, we think this is an intriguing concept and are excited to see how it evolves.