*Geek Box: Meta-Analysis
The term ‘meta-analysis’ first entered scientific terminology in 1976 in a paper by psychologist Gene Glass. The intended outcome of a meta-analysis is quantitative precision; to obtain a strong statistical summary of the effect size for a given exposure and outcome. Meta-analysis can also be used to compare and contrast results from different studies, examining the reason for divergent effect sizes, and also to identify patterns between studies, for example whether there is effects at specific doses, but not others.
The first aim – to obtain a statistically precise overall summary of effect – is effective when certain assumptions are met in included studies: precise independent effects, specialised population group, clearly defined intervention, similar effect size measures. This tends to reflect the desired aim within evidence-based medicine for a meta-analysis to include evidence from RCTs, with similar methodology, looking at the same exposure-outcome relationship, for example statins and heart disease.
This is reflected in the position of meta-analysis at the top of the pyramid hierarchy of evidence, with the implication that if RCTs are to be considered the ‘gold standard’ of evidence, meta-analysis are widely considered the ‘platinum standard’. Indeed this may be the case when the exposure is a drug. However, the conceptual basis and underlying assumptions favour biomedical RCTs, and applying this methodology to nutrition – either interventions or prospective cohort studies – without consideration of these factors may often yield misleading answers.
In biomedical sciences, we may see a meta-analysis of a specific drug, like the example of statins and heart disease. But imagine doing a meta-analysis on heart disease risk and including studies on statins, niacin, aspirin, blood pressure medications, beta-blockers, fish oils, CETP-inhibitors, and assuming they were all equivocal: this is exactly the kind of “distortive lumping” that is all too typical among nutrition meta-analysis. Studies may differ in the foods or supplement used, the dose, the duration of the intervention relative to the time-course of the disease, the population group, or the outcome measures. So, with nutrition meta-analysis, we need to take extra caution in analysing their findings, and not merely assume that because of their position on the hierarchy of evidence, that they are somehow infallible.