2017 Prediction Accuracy

(This is preliminary, since a few of the data points aren’t in, and I need to check what Scott over at SSC says on a few prediction resolutions which I left to his judgement.)

Brier Score: 0.108
Log Score: -0.348

As with last year, I was slightly under-confident based on the results, but I got lucky on a few of these. (Of course I did, that’s the way predictions under uncertainty work!)

Of [50–60%) predictions, I got 2 right and 3 wrong, for a score of 40%
Of [60–70%) predictions, I got 8 right and 3 wrong, for a score of 73%
Of [70–80%) predictions, I got 6 right and 2 wrong, for a score of 75%
Of [80–90%) predictions, I got 11 right and 1 wrong, for a score of 92%
Of [90–95%) predictions, I got 9 right and 0 wrong, for a score of 100%
Of [95–99%) predictions, I got 11 right and 0 wrong, for a score of 100%
Of [99+%] predictions, I got 4 right and 0 wrong, for a score of 100%

GDP Growth: (I’m using 2.2% — the number isn’t anywhere near final, but it’s good enough to score these with.)

< 4% — 90%
lower than in 2015: 60%

No: <2% — 40%
<0% — 5%

Yes: Dow Jones will not fall > 10% this year: 80%

Domestic US Politics:

Yes: Repeal of Obamacare, including the individual mandate OR minimum coverage rules — 80% (End of year:I got lucky on this one — the December tax bill repealed the mandate.)

No: Conditional on above, Republicans have replacement policies already passed — 5% (End of year: I think I was overconfident here, but I was right.)

International Relations:

These are basically just Slatestarcodex 2016 Predictions, 2017 redux: (Numbers to show which were skipped.)

Yes — 1. US will not get involved in any new major war with death toll of > 100 US soldiers: 90%
Yes — 2. North Korea’s government will survive the year without large civil war/revolt: 98%
Yes — 4. No terrorist attack in the USA will kill > 100 people: 95%
Yes — 5. …in any First World country: 80%
Yes — 7. Israel will not get in a large-scale war (ie >100 Israeli deaths) with any Arab state: 95%
Yes — 9. No interesting progress with Gaza or peace negotiations in general this year: 95%
Yes — 13. ISIS will control less territory than it does right now: 90%
Yes — 15. No major civil war in Middle Eastern country not currently experiencing one: 65% (End of year: Iran’s getting hot, though…)
Yes — 18. No country currently in Euro or EU announces plan to leave: 95%
Yes — 35. No major war in Asia (with >100 Chinese, Japanese, South Korean, and American deaths combined) over tiny stupid islands: 99%
Yes — 45. SpaceX successfully launches a reused rocket: 80%
Yes — 48. No major earthquake (>100 deaths) in US: 99%
Yes — 49. No major earthquake (>10000 deaths) in the world: 80%


Yes — Bitcoin will end the year higher than $1000: 40% (Begin year note: But last year I was surprised — maybe I’m not well calibrated here. Or maybe something about markets, irrationality, and solvency. End year Note: No, I’m badly calibrated.

Yes — Lightning Networks / Segwit will be deployed: 65%

No — There will be a major bug found and exploited in LN/Segwit: 60%

Yes — Ethereum will be above $10: 70%

Yes — Ethereum will be above $20: 30%

Yes — Z-Cash will be above $50 (Price on 1/3): 25%

Yes — Z-Cash Market Cap will be above $18m (Value on 1/3): 75%

Yes — There will be no war involving more than one of the US, Russia, China, or any member of the EU on different sides started in 2017 (Measure: Total eventual casualties above 100) — 98%

Longer Term Predictions — No Scores:

There will be no war involving more than one of the US, Russia, China, or any member of the EU on different sides started before 2020 (Measure: Total eventual casualties above 100) — 95%

The Republicans will maintain control of the House of Representatives in 2018 elections: 40%

There will be a Republican primary challenger getting >10% of the primary vote in 2020 (conditional on Trump running) — 70%

The stock market will go down under President Trump (Conditional on him having a 4 year term, Inauguration-Inauguration) — 60%

Slatestarcodex 2017 Predictions:

(Added 1/9/2017. I have left out ones I already forecast above, from last year. His estimate in Parentheses)

Yes — 5. Assad will remain President of Syria: 90% (80%)
Yes — 7. No major intifada in Israel this year (ie > 250 Israeli deaths, but not in Cast Lead style war): 70% (80%)
Yes — 9. No Cast Lead style bombing/invasion of Gaza this year: 75% (90%)
(?) Waiting for SSC Judgement, but yes — 10. Situation in Israel looks more worse than better: 85% (70%)
Yes — 11. Syria’s civil war will not end this year: 80% (60%)
Yes — 13. ISIS will not continue to exist as a state entity in Iraq/Syria: 65% (50%) (Wikipedia: “it controlled no meaningful territory by November 2017”)
Who Knows — 15. Libya to remain a mess: Unclearly defined — 50% to 100% (80%)
I was right, but unclear score — 16. Ukraine will neither break into all-out war or get neatly resolved: [No guess — unsure if a cemented status quo with recognition is “resolved” — which seems likely.] (80%)
Yes — 17. No major revolt (greater than or equal to Tiananmen Square) against Chinese Communist Party: 99% [ China doesn’t let things get to that point any more. They intervene before it can get that large; it’s either toppled gov’t, or nothing like this.] (95%)
Yes — 19. No exchange of fire over tiny stupid islands: 80% (90%)
(?) — 20. No announcement of genetically engineered human baby or credible plan for such: 90% I don’t follow this, so I’m mostly relying on Scott’s estimate (90%)
No — 21. EMDrive is launched into space and testing is successfully begun: 50% [No date announced, and there is a launch backlog for cubesats in the US. These things don’t happen quickly.] (70%)
Yes — 22. A significant number of skeptics will not become convinced EMDrive works: 95% — if it launches, this will become clear quickly. (80%)
Yes — 23. A significant number of believers will not become convinced EMDrive doesn’t work. [If it launches, it’s game over.] Conditional: 10% if launched before November, 90% if not. (60%)
No — 26. Keith Ellison chosen as new DNC chair: 80% (70%)

Yes — 27. No country currently in Euro or EU announces new plan to leave: [Exclusing England, which has no actual plan, 90%](80%)
Yes — 28. France does not declare plan to leave EU: 97% (95%)
Yes — 29. Germany does not declare plan to leave EU: 99% (99%)
Yes — 30. No agreement reached on “two-speed EU”: 90% (80%)
Yes — 31. The UK triggers Article 50: 80% (90%)
Yes — 32. Marine Le Pen is not elected President of France: [Conditional on her running] 50% (60%)
Yes — 33. Angela Merkel is re-elected Chancellor of Germany: 70% (60%)
Yes — 34. Theresa May remains PM of Britain: 80% (80%)
(?) Yes — 35. Fewer refugees admitted 2017 than 2016: [to Europe? Unsure. But the flow is slowing, it seems. 95% (95%)

Yes— 37. Oil will end the year higher than $50 a barrel: [Brent? I guess.] 70% (60%) (It dropped, but then came back.)
No — 38. …but lower than $60 a barrel: 50% (60%)
Yes — 39. Dow Jones will not fall > 10% this year: 60% (50%)
Yes — 40. Shanghai index will not fall > 10% this year: 65% (50%)

Yes — 41. Donald Trump remains President at the end of 2017: 95% (90%)
Yes — 42. No serious impeachment proceedings are active against Trump: 90% (80%)
No — 43. Construction on Mexican border wall (beyond existing barriers) begins: 50% (80%)
Yes — 44. Trump administration does not initiate extra prosecution of Hillary Clinton: 95% (90%)
No — 45. US GDP growth lower than in 2016: 50% (60%)
No — 46. US unemployment to be higher at end of year than beginning: 40% (60%)
Yes— 47. US does not withdraw from large trade org like WTO or NAFTA: 80% (90%)
Yes— 48. US does not publicly and explicitly disavow One China policy: 80% (95%)
Yes— 49. No race riot killing > 5 people: 80% (95%)
No — 50. US lifts at least half of existing sanctions on Russia: [How do you measure half? Also, congress does this, not president. So…] 60%[?] (70%)
Yes — 51. Donald Trump’s approval rating at the end of 2017 is lower than fifty percent: 90% (80%)
Yes — 52. …lower than forty percent: [It’s only 41% now, and they typically drop — and the big exception is G.W.B., after 9/11.] 75% (60%)

And I’m not predicting his blog traffic or his life. But…
(?) 60. Less Wrong renaissance attempt will seem less (rather than more) successful by end of this year: [I’m a bit hopeful — but I’ll follow his judgement on what happened…] 80% (90%)

(?) Bonus: Scott’s accuracy / calibration will be about as good as it was last year, not materially worse [I’m gonna eyeball this one, I can’t think of a good metric.]: 90%


Evidence as Rhetoric — Normative or Positive?

I recently saw an interesting and disturbing paper, “Reframing Evidence Synthesis As Rhetorical Action in the Policy Making Drama.” To vastly simplify and rephrase, they note that rational updating based on evidence is corrupted by policymakers presentation of filtered evidence. Because of this, policy discussion based on evidence is rhetoric, not logic — and it should be viewed in those terms. You might guess from the title that I agree this is a correct description of policy making, but an incorrect and damaging way to make policy.

I want to lay that argument out more fully, including a discussion of bounded rationality, the norms of policymaking, signalling via advocacy of evidence, and the ways in which this problematic norm should be fixed — but for now I think my point is clear.