— Yung En Chee, University of Melbourne

[Editor's note: These ideas were prepared by Yung in support of her participation in a forecasting project and describe techniques she found helpful for predicting specific world events within specific time frames. The context is one where specific possible outcomes of political, economic, etc. events were described, and probability ranges for those outcomes had to be supplied within a few days. The requirement was to provide predictions as accurate and precise as possible without succumbing to over- or under-confidence. I think these ideas are of general interest.]

My list of tools and strategies is a mash-up of ideas from other people, including Daniel Kahneman (Thinking Fast, Thinking Slow), a couple of bloggers who were on the Good Judgement team - ‘Dart-Throwing Chimp’ (Jay Ulfelder) and ‘Morendil’ (Laurent Bossavit) and personal reflection. See the following Morendil posts on lesswrong:

• http://lesswrong.com/lw/etl/raising_the_waterline/
• http://lesswrong.com/lw/eup/raising_the_forecasting_waterline_part_1/
• http://lesswrong.com/lw/ews/raising_the_forecasting_waterline_part_2/

Bossavit's tools for prediction:

1. Favour the status quo (this seems to me more like an empirically derived tip – if you do this you’ll come out ahead in the long run)
2. Flip the question around
3. Use reference classes
4. Prepare lines of retreat (what would make me change my mind about this?) [this is equivalent to think of reasons/evidence that make it unlikely – a sort of correction for overconfidence]
5. Abandon sunk costs
6. Consider your loss function (this is more about strategic hedging in response to how the Brier score is computed)

These ideas are, in part, about the psychological aspects of making the prediction, but he doesn’t discuss strategies for finding information, reconciling or evaluating information, forming mental models etc.

Tools I use:

1. Understand the conditions required for resolution of question.
2. Iteration (rapid prototyping). Search and scan articles quickly to build up some mental model of what the situation is, who the actors are and understand any processes required for resolution of the question (point 3 below) – go quick, build a crude mental model then refine as more material is encountered/assimilated.
3. Understand any processes required for resolution of question. E.g. standard operating protocol for declaration of disease status of OIE member (World Organization of Animal Health); process of granting of EU member candidacy; voting and veto powers re UN Security Council, process for obtaining full negotiating member status at TPP (Trans-Pacific Partnership), etc.
4. If a question involves non-English speaking players make sure to search beyond the elite news sources (e.g. BBCNews, Guardian, Newsdaily, New York Times, Economist, WSJ) for primarily English speakers (e.g. Al-Jazeera, Al-Monitor, Kurdistan news, Ahram online etc.).
5. Consider whether the uncertainty associated with the question is reducible or irreducible. Uncertainty may be reducible if it’s due to a lack of knowledge or understanding. If I suspect this is the case, I try to find a knowledgeable interpreter/analyst or try to construct Reference Classes and identify plausible Base Rates. Indicators of knowledge: knowledge of historical context, logical argumentation, ability to articulate reasons, provide reasons supported by evidence, ability to recognise and acknowledge where and when things aren’t known (e.g. Aurelan George Mulgan on Japan TPP; Reidar Visser on Iran). If the uncertainty is irreducible (and I consider things like oil, stock and currency prices to fall in this category) then don’t bother looking for more info.
6. Use Reference Classes, try to estimate Base Rate from any available data and use that (see Kahneman: Thinking Fast, Thinking Slow).
7. Use Bueno de Mesquita’s ‘factors’ (Position, Influence, Salience, Flexibility/Resolve, see below) to think through the interests of actors involved.
8. Understand the status quo and consider favouring the status quo (empirically, we would expect the status quo to persist).
9. Another empirically robust finding from interest group politics: well-organised private interests nearly always win out over diffuse, generally disorganized public interests.
10. Try and articulate Reasons that would make an event LIKELY and and Reasons that would be it UNLIKELY (this requires adequate research or analysis or reasoning to furnish or construct reasons; will often rely on tally for informing prediction).
11. Keep looking out for new info till just before the deadline and update estimates accordingly.
12. Consciously consider our tendency towards What You See IS All There Is (WYSIATI) – this tendency is misleading! Remember that the capacity for unpredictable, unforeseeable events or chains of events is ever present, particularly with long time lines and adjust forecasts accordingly (e.g. UN Sec Council resolution on Mali).

Note: Bruce Bueno de Mesquita is a political scientist who claims to have built a system using Game Theory for making predictions (http://www.predictioneersgame.com/) I think he overreaches and he’s been criticised for not being transparent about his methods. I agree. Nevertheless, I find his key factors for assessing actor interests, stakes, power and influence useful.

1. Relative potential influence - the relative potential ability of each player to persuade other stakeholders to adjust their approach to the issue to be more in line with the influencer’s perspective. Resources - the potential ability each player has to persuade other stakeholders to support a point of view on the issue in question. The ability to persuade may be derived from holding a position of authority, being an expert, commanding a large budget, or any other factor that makes others listen to someone.
2. Policy position - position preferred by each stakeholder on the issue, taking constraints into account. This position is not likely to be the outcome the stakeholder expects or is prepared to accept, nor is it likely to be what the player wants in his or her heart of hearts. It is the position the stakeholder favors or advocates within the context of the situation. When a player’s position has not been articulated, it is best thought of as the answer to the following mind experiment: If the stakeholder were asked to write down his or her position, without knowing the values being written down by other stakeholders, what would he or she write down as the position he or she prefers on the issue continuum? To place a numeric value on the position, the investigator must first have defined the issue continuum. The continuum will either have a natural numeric interpretation, such as the percentage of uninsured on health care to be covered under a new policy or the analyst will need to develop numeric values that reflect the relative degree of difference across policy stances that are not inherently quantitative. It is important that the numerical values assigned to different positions (and they can range between any values) reflect the relative distance or proximity of the different solutions to one another.  An easy way to turn player preferences on an issue into numeric values is to place each player on the issue continuum you defined, locating then at the point on the continuum that reflects the policy they support. Then, use a ruler to measure how far each player is from one end of the line that reflects the range of choices. Let the left-hand end of the line equal 0. Then each other point on the line is simply its distance from 0 on the ruler.
3. Salience - assesses how focused a stakeholder is on the issue. Its value is best thought of in terms of how prepared the stakeholder is to work on the issue when it comes up rather than some other issue on his or her plate. Would the stakeholder drop everything else to deal with the issue? Would the player work on it on a weekend day, come back from vacation, etc.? The more confidently it can be said that this issue takes priority over other matters in the stakeholder’s professional life (or personal life if the issue is about a personal or family matter), the higher the salience value.
4. Flexibility/Resolve - evaluates the stakeholder’s preference for reaching an agreement as compared to sticking to his or her preferred position even if it means failing to reach an agreement.