002: What Makes a Forecast Good?

Welcome back to Predictions, the hottest newsletter about prediction markets and quantified forecasting.

Welcome back, everyone! In this week’s Predictions:

• News in Numbers: The biggest (or sometimes, not biggest) news stories of the week, from forecasters and prediction markets, selected by us.

• Opinion: Andrew and Clay offer their opinions, theories, insights, and discussions about the field of quantified forecasting, its application, and the domain of predictions markets.

• Global Guess: With Clay out in the wilderness at Yellowstone National Park, hiking to find coverage to post today’s newsletter, we decided to take this week’s geopolitics forecast off the lineup. What do you want us to forecast next week?

• Base Rate News: Highlighted news articles and opinion pieces by others about the quantified forecasting and prediction market spaces.

❤️ Don’t forget to like the newsletter. It’s like a tip, but free!

💬 Agree or disagree with something you read? Have a suggestion? Leave a comment!

🔁 If you loved the newsletter, please consider restacking it!

📧  Please share Predictions with family and friends! We even set up referral rewards through Substack 🙂

News in Numbers

📈 60% (+27) Will Hunter Biden be indicted before November 5, 2024? (Metaculus)

• US prosecutors to seek Hunter Biden indictment by Sept. 29 (Reuters)

📉 5% (-23) Will the WHO name BA.2.86 as a SARS-CoV-2 Variant of Interest before October 1, 2023? (Metaculus)

• US lab tests suggest new Covid-19 variant BA.2.86 may be less contagious and less immune-evasive than feared (CNN)

📈 45% (+25) Will Harvard University announce an end to preferential legacy admissions before 2030? (Metaculus)

• Are legacy admissions on the way out? (The Hill)

📉 8% (-5) At close of business on 20 September 2023, will the upper limit of the Federal Reserve's target range for the federal funds rate be higher than it was at the close of business on 26 July 2023? (Good Judgment Superforecasters)

• Federal Reserve officials back rate rise pause in September (Financial Times)

💬 Which style of formatting do you prefer for News in Numbers–this version or last week’s? Let us known in the comments!

Opinion

Last week we wrote about our doubts surrounding long-term forecasting and how accurate our current methods might be. That discussion got us thinking, if long-term forecasting questions are not ideal, then what makes for a good forecasting topic in general? If you’ve journeyed deep enough down the rabbit hole that is quantified forecasting, you have probably thought about how the discipline can be leveraged in high-impact areas to improve decision-making. 

While certainly not exhaustive, we came up with a short list. If you have additions to, or subtractions from this list, let us know on Twitter or in the comments section!

1. It has to be possible to write good, falsifiable questions about your subject matter. Question-writing is one the most important parts of any good forecast, so high quality questions are needed.

2. There has to be some reliable data set surrounding your subject matter. Not only will this be necessary to forecast the question, but it will also be necessary to have a good resolution criteria 

3. The subject matter must be iterative. If the subject doesn’t have regular updates (the signals for your forecast come out over time, rather than all at once) or repeatable forecastability (no spell check!) then it won’t be a reliable forecasting topic.

So what fits the bill? Geopolitics has been the primary arena for quantified forecasting over the history of the discipline but that is not the only subject matter ripe for more attention from the forecasting community. 

Last summer, Perry World House at the University of Pennsylvania published a research paper which explored one potential future for forecasting implementation: public policy.

But first, some useful context:

• In April, 2020, the U.K. government launched an aggregated forecasting platform called Cosmic Bazaar where civil servants and government officials make predictions on issues of national security and public health. 

  • In 2021, the Economist wrote that “since the website was launched…more than 10,000 forecasts have been made by 1,300 forecasters, from 41 government departments and several allied countries. The site has around 200 regular forecasters, who must use only publicly available information to tackle the 30-40 questions that are live at any time.”

Now where were we?

In light of these developments, the authors of the paper conducted interviews with policymakers and analysts from the U.S. intelligence community to determine which features of an aggregated forecasting platform “could make crowdsourced geopolitical forecasting a helpful tool…in informing U.S. national security decision-making.” The results of those interviews suggest that, when considering how to implement forecasts into the policy process, the government should consider:

1. Using explicit percentages to convey statements of likelihood and probability.

2. Creating a classified and unclassified forecasting platform that is potentially open to allies and partner nations.

3. Establishing the forecasting platform in an office that is central to the IC, such as the Office of the Director of National Intelligence.

4. Developing a forecast aggregation platform instead of a prediction marketplace.

5. Providing context for aggregated forecasts. 

6. Embedding the use of probabilities and forecasts into analytic tradecraft guidelines.

Then, in January 2022, Cultivate Labs announced the launch of INFER, an aggregated forecasting platform funded by the U.S. government and inspired by the alleged success of Cosmic Bazaar in the U.K. As opposed to Cosmic Bazaar, which is only available to employees of the U.K. and its subsidiary agencies, INFER is open to the public “in the hopes of challenging conventional wisdom, creating a shared view of the pivotal factors that influence global events, and quantifying and prioritizing risks to the U.S. Government's national security priorities.”

The platform also abides by many of the suggestions set out in paper’s checklist, including being housed in the right bureaucratic location (National Intelligence Council within the Office of the Director of  National Intelligence), incentivizing quality forecast rationales, providing explicit percentages, and choosing to be an aggregation platform as opposed to a marketplace. Questions are even derived from questions by policymakers on particular subject matters, making them inherently high-impact. 

So the question is, are these platforms our best bet right now at incorporating quantified forecasting into the policy process?

This is a diagram from the Liberal Arts Instructional Technology Services at UT Austin detailing the six steps of the policymaking process. The process typically begins with ‘Problem Identification’ although it is truly cyclical in nature. 

The research published by Perry World House suggests it is wise to interpolate crowdsourced forecasts around steps one and two, using forecasting to help both initially identify potential issues, and consider how to prioritize those issues within a broader political agenda.

The policy process could certainly benefit from a forecasting mindset when deciding which policies to advance and implement, but doing so from the inside may not be the easiest or more efficient method. Further research should be conducted into launching think tanks, interest groups, or lobbying firms that support forecasting as a critical addition to the policy process. Often these groups garner more influence in the policy sphere than analysts in the intelligence community, for better or worse.

But as it pertains to using forecasting as a method for improving decision making, policy is perhaps the ‘Goldilocks’ area – it is at the sweet spot of iterative events, ample data, and high impact. 

An interesting concept to facilitate this would be for someone to create a platform similar to Dune but for prediction markets. Allow users to create dashboards using aggregated forecasts from multiple forecasting platforms. Platforms would be incentivized to open access to their data because this platform would essentially be free marketing for them. 

Users could track key questions across platforms and build custom dashboards for either their own personal use, or professional purposes. This sort of flexible solution for prediction market data would also likely be appealing to governments as they would be able to use it as they saw fit. Plus, prediction markets on the blockchain already have publicly available data! If you’re interested in a concept like this, reach out to us.

Global Guessing (Updates)

Last week we forced there was a 67% chance that there will be a military coup in another francophone country in Africa before the end of 2023. For any new readers, the forecast can be viewed live on our Patreon where we will record each prediction we do for the newsletter. We also said we would be watching a few key areas that might impact our forecast in the coming months. So let’s do a quick update:

1. ECOWAS

   1. ECOWAS has been focusing primarily on Niger over the last few months, imposing sanctions on the junta-ruled country and threatening to mobilize standby forces which they did in the ousting of Yahya Jammeh from The Gambia in 2017. There are some claims, however, that ECOWAS may not have the legal backing to go through with military intervention in Niger. And ECOWAS’ relatively slower response to Gabon signals a capacity bottleneck that may encourage other countries to follow suit.

2. Regional winter elections

   1. The elections have not taken place yet so we cannot comment on them, but pay especially close attention to the DRC’s general election on December 20th.

3. Rwanda and Cameroon (given their recent military purges)

   1. Both Rwanda and Cameroon do not seem to be undergoing any particularly notable political distress that might catalyze unrest in the region. But given their military purges, they have to be countries that we track for this forecast.

4. France’s on-going response

   1. It seems as though France is planning to “withdraw most of its drones and aerial reconnaissance assets from Niger.” They also may be reducing their military personnel in Niger, some of whom have been confined to their barracks since the coup. The junta have no interest in working with French forces on their anti-jihadist agenda, so French forces have become unnecessary in the region. There is speculation some might be redeployed to Chad. French concession to junta demands could also be a positive sign for another coup this year.

5. Central Africa Republic, Burundi, and Chad

   1. Chad has had a rough go of it, sandwiched between Sudan, Niger, Libya, among its may neighbors. And the turmoil in Sudan is only making the situation in Chad more precarious. Meanwhile in the Central African Republic, Wagner continues to encroach on institutions and freedoms that could be the foundation of unrest. We couldn’t see much change in Burundi’s political situation.

Base Rate News

The Case Against Prediction | Strange Loop Canon | 09.04.23

🥩 The Meat: Rohit Krishnan argues that “an obsession with prediction implicitly overvalues a certain way of making decisions, and one that is inconsistent with how we actually take action.”

🥔 The Potatoes: We feel that Rohit is conflating prediction markets themselves with their current manifestation as public access websites. A prediction market including the top physicists in the U.S., AUKUS, and Five Eyes would likely have been a more reliable indicator of LK-99 replication likelihoods than open access PMs were this time around.

LK-99 as a cautionary tale for prediction markets | The Intrinsic Perspective | 09.07.23

🥩 The LK-99 event of 2023 put a spotlight on how prediction markets are simply speculative markets and are just as likely to display extreme volatility. And this author wants to “see prediction markets do more than just track viral tweets.”

🥔 See today’s op-ed. We’re actively concerned with making forecasting and prediction markets more useful and more practical for high-impact work. Also, we went to high school with Alex Kaplan lol. 

Manifold Market’s Inaugural Forecasting Conference—Manifest 2023—is this month.

🥩 Manifold Markets is hosting its first ever forecasting conference in Berkeley, CA later this month. Speakers will include political streamer Destiny, 538 founder Nate Silver, Kalshi CEO Tarek Mansour, and Futarchy creator Robin Hanson. Oh, and us!

🥔 This is a massive moment for forecasting. We got started in space three years ago, and even back then it was very different. There were far fewer people actively building in cool and innovative ways. But today, the space looks so different. Refreshed. We can’t wait to see first hand how far this space has come, and hopefully meet some really cool forecasting minds along the way!

Q&A with Cultivate: A look into our forecast question development process | Cultivate Labs | 06.29.23

🥩 Cultivate Labs published a Q&A with team members about how they develop questions for the prediction market platforms they create for clients. 

🥔 This might become required reading for anyone interested in forecasting. It even references an INFER article from last year that provides a detailed breakdown of strategic question decomposition. This is just corner piece reading for any forecaster.