Citation: Kwok, C.S., Boekholdt, S.M., Lentjes, M.A., Loke, Y.K., Luben, R.N., Yeong, J.K., Wareham, N.J., Myint, P.K., and Khaw, K.T., Habitual chocolate consumption and risk of cardiovascular disease among healthy men and women. Heart, 2015. 101(16): p. 1279-87
Primary question: Is there a causal effect of chocolate intake on the risk of future cardiovascular events
Exposure: Chocolate consumption (in quintiles)
Outcome: Coronary heart disease, stroke, and cardiovascular disease (combination of CHD and stroke)
Summary strength of causal inference: Low Over or underestimate?: Overestimate Summary rating comments: Due primarily to the likelihood of residual confounding, and also because of the very specific sample that this sample is drawn from, the study’s results should lead us to find better ways of assessing this relationship but should not be considered on their own. Additional comments: The article also includes an updated metanalaysis of chocolate and heart disease, and all of the studies included in that analysis are likely to suffer from very similar weaknesses as the main study in the article. Primary effect size of interest: Hazard ratios for top quintile vs no chocolate consumption: Study methods structure: Observational prospective cohort Sample size: 20951 Did strength match evidence?: Yes, language matches evidence Causal language of the results: Weak Strongest phrase: In this large prospective population study, higher intake of chocolate up to 100 g/day was associated with a lower risk of CVD and stroke, with stronger associations for mortality than total incidence. Comments: Severity of issues with generalizability: High severity Describe the target population for the study: healthy adult men and women living in high income country (UK) Describe where and among whom the sample was collected/recruited (state if the sample is not clearly defined): The European Prospective Investigation Into Cancer Cohort- enrolled in Norfolk UK between 1993 and 1997- say it’s population based but not clearly defined how individuals were sampled or eligibility (cites another study that describes) – sample was 99.6% white Caucasian. assessed outcomes using NHS data. Does the composition of the population sample influence the magnitude and/or direction of the results, as compared to in an ideally generalizable population?: Yes / most likely If “yes,” or “maybe,” briefly explain the generalizability concern.: The sample is from a particular region in the UK and the sample is composed of 99% white participants. They also likely share many environmental and socio-economic factors and therefore the relationship between diet and heart health may not translate to other populations whose underlying genetic composition, dietary habits, environmental exposures, and socio-demographic characteristics differ. Were analyses done to help with transporting results to other populations, such as modification assessments or subgroup analyses?: No Does the study administration and data collection environment influence the magnitude and/or direction of the results, as compared to a real-world change in the exposure of interest?: Can’t be determined If “yes,” or “maybe,” briefly explain the concern.: Severity of issues with missing data: Moderate severity If this was a prospective study, what was the magnitude of loss to follow up?: Can’t be determined If applicable, how did the authors attempt to remedy this loss to follow up?: N/A If data from cases and controls are generated from different sampling methods or populations, could control selection be inappropriately related to the exposure?: N/A If “yes,” or “maybe,” briefly explain the concern.: What is the total magnitude of missing data for the exposure, outcome, or included covariates?: 10-30% If applicable, how would you describe the missing data?: Can’t be determined If applicable, how did the authors remedy this missing data?: Complete case analysis Severity of issues with exposure measurement: High severity Was the exposure/intervention randomly assigned?: No If not randomly assigned, is there likely to be exposure measurement error/misclassification?: Yes / very likely If measurement error/misclassification is plausible, could it be differential with respect to the outcome?: Maybe / plausible Are the pathways to changing levels of the exposure of interest relevant to the effect being estimated?: No / unlikely Severity of issues with exposure measurement: Moderate severity Is there likely to be outcome measurement error/misclassification?: Yes / very likely If misclassification is plausible, is it the measurement error / misclassification likely to be differential with respect to the exposure?: Maybe / plausible Severity of issues with covariates: High severity Are there concerns about bias with respect to measurement error and/or misclassification of the covariates included?: Maybe / plausible If yes, describe potential bias due to covariate measurement error (including direction): Direction of bias is unclear, but likely is measurement error (LDL/HDL not taken after fasting, recall and social desirability bias regarding things like smoking and alcohol, etc.) How did the authors choose covariates included in the analysis?: Can’t be determined Are there variables missing from the analysis that could introduce confounding?: Maybe / plausible If yes, list unmeasured confounders as noted by the authors and the potential for bias, including direction.: whole dietary pattern If yes, list unmeasured confounders not noted by the authors (i.e. in the opinion of the reviewer) and the potential for bias (including direction): Besides those identified by the authors, which I agree are important, there is always the unobservable preference over health. Those who care greatly about their health and who engage in an overall healthy lifestyle are more likely to have lower risk of heart disease. But in terms of chocolate consumption the link is less clear: those who care more about their health may be more likely to eat dark chocolate and less likely to eat chocolate candy bars, which are combined in this study leading to an unknown direction of bias. Or they may be less likely overall to eat sweets generally including chocolate, which would lead to a positive bias on a negative effect, so (probably) an underestimate. Severity of issues with methodology: High severity What was the primary analytical method(s) used?: Hierarchical and/or longitudinal regression models Is the analytic method appropriate for identifying a causal relationship between the exposure and outcome in the context of this study?: No / inappropriate If inappropriate or unsure, briefly describe why.: What measures of uncertainty are reported?: p-values and confidence intervals Do statistical methods appropriately estimate error bounds?: Yes / very appropriate Describe any robustness checks against potential flaws that were completed, including functional form assumptions and other sensitivity analyses: Sensitivity to various covariates, use of propensity score matching instead of linearly controlling for covariates.
One or more substantial methodological issues are present in this study, particularly those that overestimate the effect size. Results should be considered hypothesis-generating, and should not be used alone in policy and practice.
CHD: 0.88 (95% CI 0.77 to 1.01)
Stroke: 0.77 (95% CI 0.62 to 0.97)
CVD: 0.86 (95% CI 0.76 to 0.97)
Causal language of the question: Weak
Generalizability comments: There is not a lot of information on the data collection procedures that would allow me to assess the potential for Hawthorne effects, etc. The article references other articles where the cohort and data collection process is described in more detail. However I doubt it is of major concern since much of the follow up is done via admin data from the NHS.
Missing data summary comments: It isn’t clear which covariates are missing so it is difficult to say if these things may be missing at random or not or what was going on with missing exposure assessment (857 people)– small degree of missing data on exposure makes it seem possible this was just missing at random. Because reasons for missing data not clear, direction of bias is not clear either. They did exclude 86 people with very high chocolate intake, but this is overall such a small percentage it likely doesn’t matter much. Also, it is worth noting that individuals could have emigrated out of the country and therefore, none of their outcome would be detected by their measurement mechanisms and thus, no outcomes are essentially assumed for these individuals.
Summary comments for exposure measurement: Concerns over recall for past year consumption (Mention recall bias more common among women and obese) and they measure chocolate consumption via “”singles or squares, snack bars, or cocoa/hot chocolate”” and by summing weights of those foods (frequency categories * portion size) not flavonoid or cocoa content. This will likely (a) miss some chocolate consumption and (b) misclassify the amount since cocoa content will be drastically different by type of food eaten. Could be that people with overall poorer health consume chocolate via different means due to less nutritional diet which would make misclassification related to outcome potentially; also, could be that people change chocolate consumption through dieting and that could impact CVD (maybe because have early CVD even i.e. not yet had a heart attack but have atherosclerosis)- maybe most likely is people with risk factors for poor health (obesity) will underreport chocolate intake- leading to a negative bias away from the null.
Summary comments for exposure measurement: Rely on records review (hospital and vital statistics)- which means people have to actually seek care and be recorded as having CVD (MI or stroke). They cite an earlier study where they said their case ascertainment strategy had high specificity, but did not speak about sensitivity (may be missing many cases) and ICD codes may not be consistently applied across hospitals/doctors offices etc. It could be differential if people who eat a different diet (healthier, maybe including less chocolate) are more likely to seek care and be diagnosed– this would be a negative bias and bias away from the null, but could also be that they are getting only the most severe cases and more severe cases occur more commonly in people with poorer nutrition who may consume more chocolate which would bias in the opposite direction.
Summary comments for covariates: Selective underreporting of snacks and sweets by women and those with obesity: compares accurate (higher) reporting among the healthy to under-reporting among the sick, causing an overestimate of the effect
– Overall dietary patterns: direction unknown
Summary comments on methodological issues:
Despite their ability to control for many variables and to measure outcomes later in time instead of just cross-sectionally, there is still a lot of potential for confounding. The exposure (chocolate in diet) is time-varying, which means that confounding is likely time-varying as well. An ideal investigation would have utilized repeated measurements of exposure and outcome on the same individuals over time and should have used statistical methods to control for time-varying confounders.