Notes on Constructing the Audit These notes

Design of the audit sample – Expanded SampleSampling from GPRE ARTICLES (set E) We sample from the universe of newspapers a subset of articles which is sufficiently large to include articles that are likely to be GPR=1 articles. This is set E, and contains about [ 15% ] of the articles in sample U. The ratio E/U is the GPRE index.
The subset is constructed sampling articles that contain any of these 4 words (roots): military OR war OR geopolitical OR terroris*.

We code these articles as GPR=1 or GPR=0, as follows:
1 = the article contains references to high or rising geopolitical risks.
0 = the article contains no references to geopolitical risks.<br>

Reading the Articles We select a sample of 50 out of [384] months.
For each of the selected months, working with a team of RAs, we extract [50] random articles from set G, and [50] random articles from set E.
To ensure randomness, we pick the first [50] articles in the given sets (G and E) that also contain the generic stopwords “a” and “the” and “and” (http://www.ranks.nl/stopwords)
We exclude the Financial Times from the searches since no full text can be saved for replicability purposes.<br>

How to Code ArticlesGeneral Principle If the article discusses or highlights recent past, or current or future expected geopolitical risks, terror risks, war risks, terror acts, or current wars, label it as 1.

Example: https://search.proquest.com/docview/1748535069?accountid=39704
A nightmare scenario in the Mideast, The Washington Post, Dec 14, 2015: “.. The wars in Syria and Iraq and jihadist attacks in the West have obscured yet another Middle East threat: the possibility that slowly escalating violence between Palestinians and Israelis will destroy one of the few remaining zones of relative tranquility between Morocco and Iran”. This article is coded as 1.<br>

How to Code ArticlesGeneral Principle (Continued) If the article discusses declining tensions, label it as -1.

Example of article coded as -1
“In East Europe, stunning change”: Dramatic change swept Eastern Europe in 1989, first gathering force in Poland and Hungary, then spreading to East Germany and Czechoslovakia as the Soviet Union continued to loosen the iron grip with which it has dominated its neighbors since the end of World War II.
https://search.proquest.com/docview/282685288?accountid=39704,
Chicago Tribune, Dec 27, 1989<br>

How to Code ArticlesBooks and Reviews If the articles does not highlight any of these risks, including book or movie reviews except when highlighting risks associated to them, label it as 0.

E.g. “Books of The Times; Black Military History in the U.S.: No Longer the Untold Story”, New York Times, Dec 23, 1989, https://search.proquest.com/docview/427465416?accountid=39704
coded as 0 since it mostly covers Vietnam’s war

E.g. “Books of The Times; A Nuclear Pragmatist Offers Hope”, New York Times, Dec 15, 1988, https://search.proquest.com/docview/427011256?accountid=39704,
coded as 1 as the review discusses the author’s assessment of current nuclear risks.<br>

How to Code ArticlesObituaries/Death Notices* The GPR algorithm excludes articles found in a publication’s obituaries section. Occasionally, newspapers publish obituaries for certain high-profile figures in other categories reserved for news.

If the articles does not highlight any of these risks, including obituary, except when highlighting risks associated to them, label it as 0.

E.g. “Major-General Derrick Wormald, Obituary”, The Times; London, Apr 5, 1994, https://search.proquest.com/docview/318127277?accountid=39704
coded as 0 as it recounts a death with little/no geopolitical significance.

E.g. “The World; Imam’s Death Puts A Region On Edge”, The Times; London, Apr 5, 1994, https://search.proquest.com/docview/422126099?accountid=39704
coded as 1 as the reported individual’s death instigates regional strife.<br>

How to Code ArticlesHistorical Accounts and Anniversaries If the articles does not highlight any of these risks, including anniversaries/historical accounts except when highlighting risks associated to them, label it as 0.

E.g. “Moments That Make History”, The Times; London, Dec 31, 2001, https://search.proquest.com/docview/318584735?accountid=39704
coded as 0 as the article historicizes 9/11 without discussing current developments.

E.g. “Analysis: Death From Afar, There‘s A Long History Of Us Military Mistakes. They Destroyed A Cambodian Town Like That In 1973”, The Guardian, Apr 21, 1999, https://search.proquest.com/docview/245380083?accountid=39704
coded as 1 as it assets the connection between current and past conflicts.<br>

How to Code ArticlesTensions and Markets Note: If using the -1/0/1 scale, only code articles -1 if they discuss easing tensions, not if they discuss market strength despite tensions:

E.g. “Report: Stocks Leap as Fear Over Ukraine Eases”, Wall Street Journal, Aug 19, 2014, https://search.proquest.com/docview/1553962294?accountid=39704
coded as -1 as it explicitly discusses the decline of geopolitical risk.

E.g. “Market Roundup; S&P 500 reaches 2,000, falls back”, Los Angeles Times, Aug 26, 2014, https://search.proquest.com/docview/1555923120?accountid=39704
coded as 0 as it does not discuss the decline or presence of current geopolitical risk factors. (it discuss past geopolitical risks)

E.g. “Global turmoil fails to unsettle markets”, The Daily Telegraph, Aug 25, 2014, https://search.proquest.com/docview/1555592597?accountid=39704
coded as 1 as it discusses the presence of current geopolitical risk.<br>

How to Code ArticlesWar or Military or Terror Trials Terror trials or war trials are counted as 1 if their account highlights current or recent geopolitical, terrorist or war risks

E.g. “C.I.A. Head Sees More Spy Cases Ahead”, The New York Times, Apr 20, 1994, https://search.proquest.com/docview/429703118?accountid=39704
coded as 0 as it does not discuss whether the trials have geopolitical implications.

E.g. “For Cambodia, It‘s Time To Look Ahead--And Back; Elections, Tribunal Stir Up Tensions”, Chicago Tribune, Jul 7, 2003, https://search.proquest.com/docview/419924015?accountid=39704
coded as 1 it discusses the geopolitical impact of a series of trials.<br>

How to Code ArticlesMeetings or Talks Articles discussing constructive meetings to end wars, to end terrorism should be coded as 0 unless they make explicit references to ongoing tensions or to the risk that this goal will not be achieved.

E.g: ”U.S. To Monitor PLO Pledge to End Terrorism” Boston Globe, Dec 19, 1988, http://search.proquest.com/docview/294491173?accountid=39704
coded as 0 because it does not explicitly reference ongoing tensions.

E.g: ” U.S. Presses Mideast Missile Talks” Washington Post, Dec 28, 1988, https://search.proquest.com/docview/307076045?accountid=39704
coded as 1 because article discusses tensions surrounding talks.<br>

How to Code ArticlesAppointments, Elections and Nominations Appointment or reappointment to a military position or civilian oversight of military position (e.g. Secretary of State) should be counted as 0 unless the article discusses how the appointment brings or ignites new or renewed geopolitical tensions.

E.g. “Bush's Selections for the United Nations, the C.I.A. and Top Economic Posts”, New York Times, Dec 7, 1988, https://search.proquest.com/docview/427021759?accountid=39704
coded as 0 as it does not discuss the geopolitical impact of the appointment.

E.g. “For Cambodia, It‘s Time To Look Ahead--And Back; Elections, Tribunal Stir Up Tensions”, Chicago Tribune, Jul 7, 2003, https://search.proquest.com/docview/419924015?accountid=39704
coded as 1 it discusses the geopolitical impact of a series of elections.<br>

Construction of audited indices Based on the audits, we construct the following indices:

The GPRA index is the GPR index (G/U) times the fraction of audited articles in set G in each month that are coded as 1 (GPR_AC/50).

The GPREA index is expanded GPR index (E/U), times the fraction of audited articles in set E in each month that are coded as 1 (GPRE_AC/50).<br>

Instructions for Extracting and coding articles Navigate to http://search.proquest.com/newsstand/commandline?accountid=39704
Type in the command line search the search query (see next slide for details)***
Select 50 items per page
Save full text of articles in pdf (including index), rename it as YYYY_MM and save
Save excel file of the results listing, rename it as YYYY_MM (make sure you’ve cleared selection from previous month, otherwise you will save twice as many articles)
Report coding results in column G, and any possible comments in H (mark coding results with initials)
Save edited excel file as YYYY_MM_FL, where F and L are first and last name<br>

Validation About 12.7% of all articles belong to GPRE, the sample of articles containing references to either war, terror, military or geopolitical.

Of those, based on human reading of a sample of [ 1,200 ] articles in GPRE, 64.5% are articles mentioning high or rising risks, so one can reasonable conclude that about 8% of newspaper articles contain references to high or rising war, terror, or geopolitical risks.

About 0.3% of all articles belongs to GPR (a subset of GPRE). Of those, a fraction equal to 87.3% discusses high or rising risks. Of the remaining articles, about 40% discuss “declining” tensions, rather than something unrelated to geopolitical risks.<br>

Validation The correlation matrices are as follows





There is a high correlation between GPR and GPRA, at 0.99, thus suggesting that false positives are not a problem. False positives are [12.7]% of all articles in the GPR set.
There is a low correlation between GPRE and GPREA, at [ 0.53 ], thus suggesting that there is more noise when the search is very broad. The fraction of articles in GPRE that belongs to GPREA is 64.5%, hence false positives are a bigger problem when the search is broad.
There is a low correlation between GPR and GPRE (68%), thus suggesting that naïve search returns results that are different from a detailed search.
There is a higher correlation between GPR and GPREA (82%) than between GPR and GPRE, thus suggesting that a sophisticated search is more likely to capture the true underlying measure of articles on geopolitical risks, even though it does not capture ALL articles mentioning geopolitical risk.<br>

Validation: Excluding Some Words We strive to include in the computer-generated index words that are highly likely to be used when geopolitical tensions are high or rising.
As a result of our audit, we can select all articles with false positives, and search the text of these articles for patterns in the words.
Comparing true GPRC=1 with GPRH=0 articles, the GPRH=0 articles contain more often the following words: Books, History, Museums, Art, Kennedy, Nixon, Movies (or Films), [ add other words ]<br>

Instructions for Extracting and coding articles *** The newspaper code is, depending on the year (Financial Times removed)
1984 – 1994:
pub.Exact("The Washington Post (pre-1997 Fulltext)" OR "The Globe and Mail" OR "Boston Globe (pre-1997 Fulltext)" OR "New York Times" OR "Wall Street Journal" OR "Chicago Tribune (pre-1997 Fulltext)" OR "Los Angeles Times (pre-1997 Fulltext)" OR "The Guardian" OR "The Daily Telegraph" OR "The Times")
  
1995-1996
pub.Exact("The Washington Post" OR "The Washington Post (pre-1997 Fulltext)" OR "The Globe and Mail" OR "Boston Globe (pre-1997 Fulltext)" OR "New York Times" OR "Wall Street Journal" OR "Chicago Tribune (pre-1997 Fulltext)" OR "Los Angeles Times (pre-1997 Fulltext)" OR "The Guardian" OR "The Daily Telegraph" OR "The Times")
  
1997 -
pub.Exact("The Washington Post" OR "The Globe and Mail" OR "Boston Globe" OR "New York Times" OR "Wall Street Journal" OR "Chicago Tribune" OR "Los Angeles Times" OR "The Guardian" OR "The Daily Telegraph" OR "The Times")

The search query is, in August 2014, for example
pub.Exact("The Washington Post" OR "The Globe and Mail" OR "Boston Globe" OR "New York Times" OR "Wall Street Journal" OR "Chicago Tribune" OR "Los Angeles Times" OR "The Guardian" OR "The Daily Telegraph" OR "The Times") AND DTYPE(article OR commentary OR editorial OR feature OR front page article OR front page/cover story OR news OR report OR review) AND(PD(Aug 2014)) AND (military OR war OR geopolitical OR terrorism OR terrorist) AND (a AND about AND above)<br>