Aim

This PhD project will establish a novel statistical framework for applying natural language processing (NLP) algorithms to text data to measure economic concepts of interest and using the outputs of these algorithms in downstream econometric analysis.

Objectives

1. Develop a framework for analysing the statistical properties of different NLP approaches, improving researchers’ ability to judge the appropriateness of a method in their context, conduct valid inference (defined as ensuring unbiased and consistent estimation and constructing correct confidence intervals), and communicate their findings to stakeholders.
2. Apply the framework to a specific research question in economics (to be identified during the first year of the PhD).
3. Produce a step-by-step process for using this framework.
4. Create programming packages in R and Python for easy implementation.

Current challenges

NLP methods are used across economics, but they are applied in the absence of any

formal statistical guarantees. There are two key challenges for economists using text data:

• Performing valid inference

In general, researchers treat the upstream NPL measurement task and the downstream econometric analysis separately. This creates problems for statistical inference, as the downstream econometric model does not account for uncertainty in the text algorithm output, and it also ignores any statistical relationships between the downstream variables and the text data. Both these factors may add measurement error and create bias in the downstream model.

• Choosing the best NLP approach

A researcher has many options when it comes to which method to use to measure their concept of interest. Each method creates different results, which can impact downstream econometric analysis, producing meaningful differences in estimated treatment effects or predictions.

The process for choosing between models in the economic literature is judgement based and is generally not as rigorous as other areas of econometrics. This can lead to inconsistent research, where differences in results between studies are driven by opaque judgements of the analyst rather underlying fundamentals of the research.

Novel contribution

My statistical framework will address both issues. My framework will jointly model both text data and downstream economic variables, providing guidance on what are the conditions for and the threats to consistent and unbiased estimation and how to construct valid confidence intervals. This statistical framework will also help researchers better judge the trade-offs in using one NLP approach compared with another (for example, given a researcher’s study context which approach is most likely to produce a consistent estimate).