“Introduction: … In this article, we focus on three major sources of bias for which there is sufficient data to verify their existence, and show how they could substantially affect vaccine effectiveness estimates using observational study designs—particularly retrospective studies of large population samples using administrative data wherein researchers link vaccinations and cases to demographics and medical history….
In each of our three illustrations, we compare results based on observational study methods against randomised controlled trial (RCT) methods. For each comparison, one side represents a published study while the other is a counterfactual. In each case, we show how the gap between observational and RCT study results is due to a source of bias…
Lessons Learned: A recent commentary discussed multiple factors that can bias estimates of covid-19 vaccine effectiveness, such as vaccination status misclassification, testing differences, and disease risk factor confounding. Our article complements these observations by providing examples based on actual data sets that quantify how case-counting window bias, age bias, and background infection rate bias can profoundly complicate the analysis of observational studies, shifting covid-19 vaccine effectiveness estimates by an absolute magnitude as high as 50% to 70%...
Our analysis shows that real-world conditions such as non-randomised vaccination, crossovers, and trends in background infection rates introduce strong, complex biases into these observational datasets. Our contribution is to size up three important biases, the magnitude of which surprised us and may surprise you. We conclude that 'real-world' studies using methodologies popular in early 2021 overstate vaccine effectiveness. Our finding highlights how difficult it is to conduct high-quality observational studies during a pandemic.”
This paper was published under Open Access; CC BY 4.0 DEED, Attribution 4.0 International