Sheena Chestnut Greitens: Welcome to Horns of a Dilemma, the podcast of the Texas National Security Review.

I'm Sheena Chesnut Greitens, the editor in chief of TNSR, and I'm here with our executive editor, Ryan Vest.

We're really pleased to have joining us today, Dr.

Jeffrey Friedman, author of the article, "The World Is More Uncertain than You Think: Assessing and Combating Overconfidence Among 2,000 National Security Officials," which is featured in Volume 8 issue 4 of the journal.

Jeff is an associate professor of government at Dartmouth College where he studies the politics and psychology of national security decision making and directs the John Rosenwald Postdoctoral Fellows Program in US Foreign Policy and International Security.

Jeff, welcome to Horns of a Dilemma.

It's great to have you on the show.

Jeff Friedman: Thanks for having me, Sheena.

Sheena Chestnut Greitens: So you make a pretty stark claim in the article that national security officials are really overconfident in their assessment of uncertainty.

What, tell us a little bit about what inspired you to investigate this phenomenon of overconfidence and what broader questions about decision making and risk and managing risk in the national security arena were you hoping to answer when you started this project?

Jeff Friedman: Sure.

I mean, the main motivation for this piece is that uncertainty is everywhere in foreign policy decision making.

So that's, you know, when a general makes a war plan, you never know exactly what's gonna happen.

Or intelligence analysts who are trying to assess other states, that's surrounded by uncertainty, trade negotiations, trying to figure out each other's bottom lines.

In foreign policy, there's always things that you don't know and you have to grapple with that.

There's also lots of reasons to believe that human brains are wired in ways that make us pretty bad at assessing uncertainty, or at the very least, expose us to lots of heuristics and biases.

And there's a whole field of political psychology that explains why that's important.

In any event, it's also really hard to know how good, or how poorly, or how biased national security officials are when they assess uncertainty.

There's a bunch of reasons why that's very hard to know.

Part of it is that national security analysis is often classified.

Part of it is that national security officials often leave assessments of uncertainty vague.

And so basically what that means is we don't have a particularly rigorous picture of just how well or how poorly foreign policy officials assess uncertainty, what kind of biases they have or how hard those are to correct.

And so the article was basically an attempt to get some traction on that and figure out how well, how poorly national security officials can assess the uncertainty that surrounds their most important choices.

Ryan Vest: You know, it might be helpful at this point to talk about why it's so hard to understand how well people assess uncertainty.

I'm really curious to know how you can assess whether any single assessment of uncertainty is right or wrong.

Jeff Friedman: Yeah, that's a great question and that's really the heart of the matter, is that it's almost impossible to know whether any one assessment of uncertainty was accurate or not.

So for example, if a general says the strategy has a 70% chance of success and it fails, it could be that the general was over optimistic, or it could be that, that they just got unlucky and that the 30% outcome happened.

So for a whole bunch of reasons, it's just very hard to know what you make of any individual prediction in world politics.

And so what decision scientists generally recommend is that rather than try to evaluate predictions or assessments one by one, you instead put them together.

And for example, you could look at all of the times when generals say that their strategies have a 70% chance of success.

You can then see whether those strategies in fact, succeed something like 70% of the time.

And even if any one of those judgments could be good or bad, and that's hard to know, you could at least draw some systematic inferences about whether those judgments on the whole are accurate or biased.

Sheena Chestnut Greitens: So it's those systematic inferences that I found really interesting and a really striking contribution in this article.

So I'm going to look at my paper here so I get the numbers right, but, you know, many of us would assume that seasoned national security professionals are highly trained and that they might have better calibrated judgments.

That's at least what many of us, I think, would hope and many citizens in places around the world would hope is true of leaders with training and experience.

But your findings actually don't really suggest that that's true.

You report that when officials gave a 90% confidence rating, they were actually only correct about 57% of the time.

Which is a pretty big gap.

So can you talk a little bit more, and I know you started to get into this, but tell us a little bit more about why that kind of gap in confidence and overconfidence might persist even among experts with really extensive experience and education, and what's at stake when somebody gets it that wrong.

Jeff Friedman: Yeah, so those data that you mentioned come from, I think the total was 60,000 assessments of uncertainty made by 2,000 people.

And that allows us to draw some pretty precise inferences like you just said about just how overconfident they are.

So another number that jumped out of the data at me is that when national security officials were totally certain that they had the right answer, that is to say they said something at a 100% or 0% chance of being true, they were actually wrong 25% of the time.

So that is a pretty big gap.

It's about the same gap you'd see when you study other populations, so-called non-elite populations.

So this isn't something that's special to national security officials, but rather that this appears to be a place where national security officials have biases, in this case overconfidence, that are shared widely in the population.

And then the question is, why that is.

And I think the straight answer to that is that most people never receive structured feedback on how well they assess uncertainty.

That assessing uncertainty is something that most people do every day of their lives, but as I mentioned before, it's really hard to get a clear sense of how right or how wrong or how accurate your judgements are.

And most people will naturally think that they're better at this than is really the case.

There's a really important essay on this by Philip Tetlock in which he shows that when forecasters are asked to say whether judgments that seem accurate reflect well or poorly on their judgments, they love taking credit for things that look good.

But when they make judgments that seem after the fact to be mistaken, they start making excuses.

They say, well, actually, the question was worded poorly, or something happened that nobody could have predicted.

So you can't fault me for not understanding that.

Or actually, I just worded it poorly, and I was really closer to the truth than it looks.

And I think that's very consistent with a suite of psychological biases that lead people to think they're more efficacious at dealing with the world than they really are.

And unless you give people direct, structured, clear feedback that shows them the extent of their biases, I think it's very easy for people, even successful national security officials, to not realize the gaps in their judgment that they've got.

Ryan Vest: Studies like yours are notoriously difficult to conduct at scale.

What makes this kind of research so challenging and how does your study build on or go beyond the previous work that's been done in this arena?

Jeff Friedman: Yeah, the scale is the key thing here, because when we're talking about trying to get access to national security officials, obviously these people are pretty hard to reach.

By contrast, it can be pretty easy to gather data from the population writ large.

But there are always questions about whether surveys among, say, college students or people online would tell us something about national security officials and their performance.

So there have been a handful of studies about how well national security officials assess uncertainty in the past.

By far, the best of them to date was conducted by a Canadian scholar named David Mandel.

He got a data set of about 1500 predictions made by the Middle East component of the Canada's Intelligence Assessment Secretariat, and he did amazing things with that and that study received a lot of due recognition.

But at the end of the day, it's a relatively small slice of data produced by a very specific office in one context.

And we can always ask questions about how well that generalizes.

So what I tried to do in this study is to conduct a survey that was large enough so that we can sustain generalizable inferences about national security officials.

The short story is that I set up a collaboration with four advanced military education institutions, the National War College in the United States, the NATO Defense College in Italy, the Canadian Forces College in Toronto, and the Norwegian Defense Intelligence School in Oslo.

And over a period of eight years, I would visit those places, administer a survey to students who were mostly rising colonels or their equivalents in civilian national security agencies.

That allowed me to get a data set of about 60,000 assessments of uncertainty that were made by about 2,000 national security officials spread over, I think more than 40 NATO members and partners.

So the overall result of this is a body of data that's very large, and crucially, that spans multiple nationalities, military, civilian, different branches, different agencies, and that allows us to draw some pretty generalizable inferences about how well, or how poorly national security officials assess uncertainty.

Sheena Chestnut Greitens: So your survey asked participants to make judgements both on current facts and on future forecasts.

And I wondered, you know, we've been talking at a pretty high level so far, but could you drop down and give us an example of what those questions looked like and how you constructed them and why you felt like it was really important to ask both types of questions, both the present and the future forecasting ones?

Jeff Friedman: Sure.

So there are a bunch of questions in the dataset that are asking people to assess current states of the world or past states of the world.

So that would be, for example, does NATO spend more on defense than the rest of the world combined?

That's just a factual question.

There's an answer to that.

Of course people don't necessarily know the answer, so in their minds it's uncertain.

But that's one kind of question.

And then there are other questions in the study that are predictions.

So what are the chances that there'll be a ceasefire between Russia and Ukraine by a certain date?

Or what are the chances that Bashar al-Assad is ousted from Syria's presidency by a certain date?

And people are there making predictions about the future.

The bottom line, it turns out, that the same kind of cognitive biases appear on both of those questions.

And the reason that's important is that in the intelligence world and in the foreign policy decision making world, people often treat those two exercises, making predictions and assessing current states of the world, as being very different things.

So for example, intelligence scholars will draw a distinction between puzzles to which the answer is noble and principle and mysteries to which the answer is not known at all.

And it's often thought that that requires different cognitive mindsets that might reveal different levels of performance.

Or for example, when I started presenting this kind of material at places like the National War College, and I would talk about studies that, let's say the Good Judgment Project had done on how well people make predictions about world politics, there were a lot of people who said that that's fair enough, that's just not what we do.

A lot of what we do is trying to assess the world as it is and not trying to speculate about the future.

So I think it's really important to show that the same kinds of biases hold for both of those domains.

Another reason this is important is that it takes a long time in order to gather data on predictions and score how well people do at predicting the future.

Whereas by contrast, you can rapidly gather and analyze data about how people respond to factual questions about the world.

And to the extent that it turns out that the way people demonstrate biases in those two different kinds of questions is pretty similar, that suggests you can get a lot of mileage out of surveys that are relatively easy to conduct, in which in a period of, in our case, 20 minutes, you can gather and process information about how well people assess uncertainty.

And you can be pretty confident that that tells you something generalizable about people's intuitions for thinking about the world that isn't limited to one particular domain or another.

Sheena Chestnut Greitens: That is really interesting.

I wanted to just follow up and ask, was there anything else about the findings that jumped out or really surprised you?

Jeff Friedman: The main thing that comes out of this is just how consistent these findings are.

And in a data set that's as large as the one that we're analyzing, there's so many ways in which you can break down the data.

So, for example, you can look at military officials versus civilian officials or US based officials versus non-US based officials.

In the end people from more than 40 NATO partners and allies are in this sample.

You can break it down by gender.

You can break down in some cases by specialty.

And basically every way you cut this data, you see the same results that people are overconfident, and that people who have more certainty in their responses tend to be less accurate on the whole.

And it just becomes very clear from the data that these are generalizable patterns.

These are elements of heuristics and biases that are, you know, work their way into our brains, and then appear to affect pretty much everybody who hasn't received systematic feedback on the issue.

And that sort of striking homogeneity of the patterns throughout the data are, I think, the most striking thing about them.

Ryan Vest: I thought it was interesting, you talked about intuition and measuring intuition, and in the article itself you emphasize that the study's findings should be interpreted as measuring the intuition of national security officials to assess uncertainty, because oftentimes quick decisions have to be made without the opportunity for deep analysis or deep thinking about them.

Why do intuitions matter in this context and what might be the limitations of this kind of approach?

Jeff Friedman: Yeah, that's an important point to raise, that essentially what I did was to ask participants at these military education institutions to take surveys in which they assessed uncertainty in response to a bunch of questions.

And it's obvious that the, the answers they gave there wouldn't reflect the care or diligence they would put into writing a national intelligence estimate or making a military plan.

That's obviously something that's different.

So when I presented this material to the National War College or the NATO Defense College, I would say, I think that these reflect your cognitive first steps when getting your thinking straight about an analytic problem or when assessing uncertainty.

And those cognitive first steps aren't everything that happens in foreign policy analysis, but they're clearly very important.

So for example, national security officials often have to make decisions under time pressure where they have to rely on their intuitions.

They just don't have the ability to conduct structured analytic techniques.

For what it's worth, a lot of times when I spoke to places like the National War College, special forces officers often engaged with the material more readily than others because they just understand they have to rely on their intuitions when they're in the field, and they need to make sure that those cognitive first steps are things that they can trust.

And then additionally, even if people take the time to conduct a structured process, there's a lot of reasons to believe that the initial intuitions people bring into those conversations matter a lot and anchor the ways that the rest of the bureaucratic process might run.

And of course we might think about ways in which we could design analytic processes or bureaucratic structures to counter those biases.

There's a lot of research that suggests that sometimes those kinds of conversations can even make things worse.

And I think we can all be confident that if we could make the inputs better, if we could make it so that the first steps people take when they approach any decision problem are more valid ways of thinking about the world, then that would be a good thing to do.

And the importance of taking those first steps correctly is why political psychology is such an important part of our field and why people like Daniel Kahneman won the Nobel Prize for identifying all the ways in which our cognition can lead us astray when we rely on our intuition to think about uncertainty.

Sheena Chestnut Greitens: So following up on that, one of the other findings that really jumped out at me in the piece is the consistent bias toward false positives— that people were much more likely to believe statements were true, when in fact they were not.

And so I wondered if you could give an example of a question where this came up that might seem particularly consequential, or kind of just talk about how this tendency showed up in the data that you gathered and why It matters.

Jeff Friedman: Yes.

So the data were embedded with a whole bunch of experiments that I ran to try to get inside people's cognition and figure out why it is that they were overconfident.

One of the things that I did was to randomize how questions were presented.

So for example, some people might get a question asking, has ISIS killed more civilians over the last five years than Boko Haram?

And then the other half of people get it the other way, has Boko Haram killed more civilians over the last five years than ISIS?

Now, in principle, your answer to one should be the compliment of your answer to the other.

So if people think there's a 60% chance that ISIS was more dangerous, they'd say there was a 40% that Boko Haram was more dangerous.

I mean, these are identical questions.

So the way they're phrased shouldn't impact the answer, and the answers to those two sets of questions should add up to a hundred percent.

But it turns out that when you add them up, they add up to 110% because basically everybody is inclined to put a little bit extra probability towards whatever outcome it is you're asking about.

So when I ask, is ISIS more dangerous than Boko Haram, people start thinking about ways in which, oh, yeah, ISIS really does seem pretty dangerous.

And when you phrase it the other way, people start imagining all the reasons why maybe the right answer is Boko Haram.

There are a bunch of reasons this could be the case.

I think that the simplest explanation is what's called the availability heuristic— that when you're asked to envision some outcome, it sits in your mind and then the easier it is to think about that thing, the more likelihood you're going to assign to it.

And that's a very well documented bias, and I think that can explain why there's this bias towards false positives.

You ask people to consider an outcome, they think It's more likely than it really is.

One of the reasons this is super important is because, as we could talk about later, there are a number of methods that you can take to reduce that.

For example, you can try to make sure that analysts look for information that disconfirms hypotheses to counteract their natural tendency to agree with whatever outcome you put before them.

Or you can try to make sure that analysts always consider multiple hypotheses at the same time.

And it's very well documented that these things work.

And one thing that the data demonstrate is that there's a real need for that, because in the absence of some analytic structure that helps to counteract people's natural tendencies towards false positives, they're likely to get a bit carried away in assigning too much likelihood to almost any scenario you put before them.

Sheena Chestnut Greitens: It's important for research design for students as well.

I'm thinking about conversations I have that you have to go look for disconfirming evidence and always ask how you would know if you're wrong, not just sort of look for the evidence that suggests you're right.

So that's a really important principle for national security decision making, but I think also for scholarship.

So really glad to raise that.

Ryan Vest: It really brings up a lot of questions though.

I was really struck reading through this, that you used Brier scores to quantify forecasting accuracy.

And in your article, you found that 96% of survey participants would've received better Brier scores if they had attached less certainty to every one of their intuitive judgments.

What does this say about the overall reliability of national security officials expert judgment?

Jeff Friedman: Yeah, that's an important finding.

I'm glad you raised it.

This is very consistent with findings that Philip Tetlock had when he interviewed international relations academics.

He found they were also so overconfident that in many cases their judgments would've received better Brier scores if they had simply said they didn't know the answer to every question posed.

I should just say that the way that the Brier score works is it sort of takes the difference between the prediction you made and the assessment you could have made if you knew the answer with certainty.

And then you fiddle around with it a bit in order to get the mathematics to work.

But it's just a way of saying whose assessments are more informative than others.

One of the things that the study found is that the average forecaster or average national security official made predictions that were less accurate than simply flipping a coin.

Sixty-eight percent of them were worse than that flipping a coin standard.

And the more certain that they were, that is the more extreme the probabilities they attached to judgments presented to them, the worse their Brier scores tended to be.

So more confident people tend to be less accurate.

And as you said, that was so widespread that 96% of people in the dataset would've done better if they had made every judgment with less certainty.

So these are really striking findings.

I think it is important to indicate that this doesn't mean that people don't know what they're talking about.

So the data are really clear that when national security officials think some statements are more likely to be true than others, that those are reliable judgments.

So if you just ask them to say, you know, in a relative sense, when they think something's more likely, it is in fact more likely to be true.

It's just they overdo it.

And they're so overconfident, they assign too much certainty that that in some sense cancels out a lot of the value that they can provide.

And the key to this is not to say that we shouldn't assess uncertainty.

It's not to say that national security officials don't have expertise.

They clearly do.

It's just that that is often counteracted by their cognitive biases.

And that's why it's so important to better calibrate these judgments to screen out the overconfidence, so that we can make most use of the information and the insight that people really have.

Sheena Chestnut Greitens: Okay, so now I've got to ask this question about human psychology and political psychology.

How did national security officials react, or how have they reacted to being shown that their judgments were so biased and that their estimates of uncertainty were so off?

Because this was a really interesting collaborative research process, and I'm curious what the impact and reception to you showing people this data was and has been.

Jeff Friedman: It was a wonderful experience to present this to these people.

So again, there were about 2,000 students at places like the National War College or the NATO Defense College, Canadian Forces College, Norwegian Defense Intelligence School, and they would take these surveys and then I'd go and present the findings to them.

I would say there was always, you know, like some nervous laughter when people saw that their judgments were so overconfident.

Generally, the room gets a bit quiet when you say people would've gotten better scores if they had just flipped a coin.

On the other hand, people were very receptive to this.

One thing that I've always found about national security professionals is they want to do a good job, they want to do better.

And I think they're also quite aware that the kind of data that they're presented with in these exercises are things that almost none of them have ever seen before.

And so I think this general idea that it's possible to go through your career and do really well, indeed well enough that you can get a really coveted slot at a place like the National War College, but yet never receive direct feedback on your ability to assess uncertainty.

And that's something you want to have, particularly if you're going to make high stakes choices in the world.

I think that's something people were on board with.

A question that just was always asked and is an important one is, how is it that we can make decisions and be confident in decisions if you're telling us that we need to grapple with uncertainty all the time?

And if you're telling us that when we are certain that things are going to work, we actually should think that there's only like a 60% chance they're going to work, you know, how can we get through our jobs?

And I think the answer to that—and it's a super important question—I think the answer to that is just to distinguish between being confident as a sense that you are certain something is going to work out as opposed to confidence meaning you are sure you have made the right choice, given the information that's available to you.

And I think quite clearly, we want national security officials defining good decisions in terms of the second of those things.

And indeed if we think we can only make confident decisions when we are certain they're going to work, that puts us in a really vulnerable position because the world is more uncertain than we think, and a lot of our assumptions are going to be proved wrong.

Whereas if we've done a really good job at assessing uncertainty and acknowledging doubts and caveats, and we can be confident we're doing those in an accurate manner, and we still think the decisions we're making are justified, I would say that should give us more confidence that we're doing the right thing.

And I just think that conceptual distinction of saying being confident in a decision is means you've thought about it well and not that you are absolutely certain it's going to work, is an important distinction that's worth drawing and always occupied a good chunk of the conversations I had with the professionals who participated in this.

Sheena Chestnut Greitens: That's great to know.

That's really well put and I appreciate the answer and the extra context.

Ryan Vest: Yeah, definitely.

Now, something that I thought was really positive about your article, you argue that with just two minutes of training, you saw significant improvements in performance on the survey.

What did that training involve and what made that so effective?

Jeff Friedman: Yeah, I'm glad you asked that.

So one of the experiments that was embedded in this study was that half of every cohort that participated would get a very brief set of information that was basically just the same statistics we've been talking about.

So when national security officials think a statement has a 90% chance of being true, it's actually true about 60% of the time.

Ninety-eight percent of people who participated in the study would've done better if they made every one of their judgements with less certainty.

And there was a nice graph that sort of summarized all this.

And it, as you said, we can look on these surveys and see exactly how long people spent looking at them, and the answer is on average two minutes.

And after receiving just that two minutes of training, the people who got that information performed way better than their counterparts.

That was statistically significant, and it's substantively significant too, it's about a quarter of a standard deviation.

So that's a lot.

And again, it's just two minutes.

And that's consistent with information from many other scholars, including the Good Judgment Project, which found that I think one hour of training in their case led to persistent and measurable improvements in people's ability to predict international politics over a period of years.

I do similar exercises every year with my students at Dartmouth.

We give the same kind of lecture I gave at the war colleges, and their performance then improves on another round.

So I think part of what this is building towards is the idea that almost everybody is susceptible to overconfidence, but it's not like this is a bias that is impossible to counter.

And it's probably there largely because people are truly unaware of just how overconfident they are.

They're unaware of it becuase they've never gotten this information before.

You go through your whole life and never receive structured feedback on how well or how poorly you assess uncertainty.

And I think the good news is the data indicate that once confronted with that information, people can smarten up pretty fast.

And one of the main implications of the study is it would potentially be a very good idea for national security bureaucracies to implement these kinds of procedures at scale.

And one of the goals in writing this article was to give people a sense of how they could do that in whatever context they come from.

Sheena Chestnut Greitens: So I thought that was a really striking implication of the findings of the article.

And, you know, you'd emphasized earlier in this conversation the importance of feedback, but yet also just how rare it tends to be.

So I wanted to ask you a little bit, and this is, this is out a little bit, going a little bit out beyond the scope of the article's empirical analysis itself, but why you think it's so rare for national security organizations to provide the kind of feedback that you were able to provide people.

And, you know, what do you think the effect of, and again, I'm asking a sort of speculative question here, but what do you think the effect of regular exposure to this kind of feedback might be over time if just two minutes can make a fairly significant correction, as a one-off.

Jeff Friedman: Yeah, it's a great question, and I mean, in terms of why this is not more widespread, honestly think the main answer is most people are just not familiar with these concepts.

I mean, I certainly didn't encounter them as an undergraduate or when I worked in Washington, or even when I went to grad school.

I just had to read about this on my time.

I think people get lots of training throughout their careers, but it's a pretty rare profession that would provide this sort of training to its personnel as a matter of course.

So I think there's no main obstacle here other than it's unusual.

It's a bit odd.

It requires you to think a little bit differently about how to assess the quality of assessments of uncertainty.

And I think the places like that I work with, like the National War College and the Norwegian Defense Intelligence School, Canadian Forces College, NATO Defense College, they deserve a lot of credit for being willing to try this.

And once they saw it, and, you know, particularly after the first time I'd show up and tell these high flying colonels how overconfident they were, I think a lot of them saw the value and wanted to do it again.

So I think, I think basically, you know, a hope for a project like this is that if you can demonstrate proof of concept and show that this is something that can be evaluated rigorously, be done in a manner that is useful to the missions of these institutions with which we're working, then maybe it would have some chance to spread.

And I think that all the research presented here and in similar studies indicates that there's reason to think that even relatively brief interventions can have a pretty measurable impact.

Ryan Vest: In your findings, you mentioned that one of the cognitive patterns that you found was this idea of single outcome forecasting where officials generate reasons why one scenario might be true, but ignore a lot of the alternatives.

Why do you think this is so common and how can it be avoided?

Jeff Friedman: Yeah, so the idea of single outcome forecasting is, you know, I just ask you to assess the chances that one thing is true, and you don't necessarily have to consider alternatives at the same time.

There are so many examples of this, but the one that often gets discussed as quite salient was that when the US intelligence community was trying to assess whether Saddam Hussein was building weapons of mass destruction, one of the key pieces of evidence they had that seemed to indicate Saddam was building a nuclear weapons program was series of aluminum tubes that the country was importing at that the United States intercepted, and seemed plausibly destined to build uranium centrifuges to enrich material to build a bomb.

In any event, there was always another hypothesis for what these aluminum tubes were going to be used for, and that was to build conventional rockets.

And it turns out that that was indeed their intended purpose.

But a lot of the analytic process that went into analyzing the subject, wasn't directly pitting those two hypotheses head to head.

It seems like in a lot of cases what intelligence analysts were doing is just sort of asking themselves, do these tubes look like they would be used to build a nuclear weapons program?

And eventually they concluded that that was the case.

So that's single outcome forecasting.

And the way you combat that is to make sure you always assess multiple outcomes, that you never write a study that just asks what are the chances that one thing is true?

Instead, you want to make sure that every analysis considers multiple hypotheses simultaneously.

You're simultaneously assessing the chances that the tubes are going to be used for centrifuges or rockets or maybe some other thing you can't even consider.

There are a number of techniques you can use for this.

I mean The standard one at CIA is called the analysis of competing hypotheses.

But I think basically just the intuition here is to understand that if you fixate on any one potential outcome, you're liable to overrate how likely it is, and that can in turn, lead to all sorts of problems that were, of course, very vividly shown in the estimate of Iraq's nuclear programs in 2002.

Sheena Chestnut Greitens: So part of our mission at the Texas National Security Review as a journal is really to try to have scholarship that speaks to policy, and it speaks to important policy questions, important national security decision making questions.

And so I, you know, I really hope that decision makers, military, civilian national security officials, intelligence analysts have a chance to read and mull over and think about your findings and would love to see them adopted or taken into account at an organization wide level.

But in the meantime, I'm curious about, you know, an individual decision maker or analyst who is listening to this conversation who wants to get better, right?

I agree with you, a lot of the people I interact with, whether it's on the academic side or somewhere in government, the US or overseas, want to do their jobs well.

And so if somebody's listening to this and they're like, gosh, you know, the world is more uncertain than I realized before I listened to this conversation.

What can I do to get better?

Are there things that you would recommend in professional military education or simply in people's own learning and training themselves that they could do that would improve the quality of their decision making and judgements, for somebody who's interested in trying to do that?

Jeff Friedman: Yeah, there are a number of things.

I mean, the main thing you want to do regardless of who you are— and this is material I teach at Dartmouth, the students, regardless of what careers they want to go into— you just have to make sure that at some point you get some direct feedback on how well or how poorly you assess uncertainty, and luckily these days there are a million ways you can do that.

You can participate in online prediction markets.

You can take a number of online instruments that will give you real time feedback on how are you're doing.

Just anybody who now wants to just like open up your browser and type in "probability calibration survey," you'll find a bunch of ways that you can do this.

You can participate in the Good Judgment Project.

You can participate in Metaculus's prediction poll.

I mean, there's so many ways in which you can do this if you want to.

The other thing that you could do if, you know, particularly if we want to institutionalize these kinds of ideas is just to have more institutions incorporate them into their curricula.

So when I was working with places like the National War College, I would speak in their leadership training curriculum, and that was something that these institutions thought was a really important way of convincing their students that there was a lot about the world that they didn't know.

And there was a reason why they needed to go back to school, as it were, in order to train, in order to make good command decisions.

And I think that any institution that wants to make those investments can do it.

I was really overwhelmed by the reception that I was given at the institutions I work with.

I'm just incredibly grateful that they were willing to go out on a limb to work with me.

And hopefully now, with all the results that are presented in the paper and having spent the better part of a decade working with these institutions to hone a program that they found useful and wanted to incorporate into their curriculum year after year, that may provide, I hope, a roadmap for other institutions who want to do the same thing.

And one of the reasons I wrote the paper is so that it can spread, and that maybe in the future, other people can conduct similar exercises on their own.

And it was just such a rewarding process that I hope other people will take up themselves.

Sheena Chestnut Greitens: That's great and it is heartening that it's had that kind of effect and reception and definitely hope that the article helps to spread the word and get it out there.

Thanks for joining us today on Horns of a Dilemma from the Texas National Security Review.

Our guest today has been Jeffrey Friedman, author of the article, "The World is

More Uncertain Than You Think: Assessing and Combating Overconfidence Among 2,000 National Security Officials." Jeff, thank you very much for joining us today.

Jeff Friedman: Thanks Sheena, and thanks, Ryan.

It was a pleasure to be here.

Sheena Chestnut Greitens: If you all enjoyed this episode, please be sure to subscribe and leave a review wherever you listen.

You can find more of our work at TNSR.org.

Today's episode was produced by TNSR Digital and Technical Manager, Jordan Morning, and made possible by the University of Texas system.

This is Sheena Chestnut Greitens and Ryan Vest.

Thanks for listening.