Patrick Doupe, Principal Economist at Zalando SE, shares his team's experience about what it takes to raise attention to causal inference in practice.
In industry we have a lot of questions that cannot be answered with A/B tests. As a community we know we could answer many of these with observational causal inference methods. As described in the above quote, if people don’t understand the methods the methods will be overlooked.
"People tend to overlook what they don’t really understand. People will only remember the number." – Product Analyst
We face a chicken and egg problem. Analysts and other producers of research don’t understand enough about causal inference to produce credible numbers. Consumers of research don’t understand enough about causal inference to interrogate and understand the numbers analysts produce. These sides reinforce each other: no incentive to learn how to do matching if a naive comparison of means will bring you the same rewards; no incentive to learn how to assess estimates if teams cannot produce credible estimates. The problem is switching to a better equilibrium.