Dissertation

A population based approach to diabetes mellitus risk prediction:

Methodological advances and practical applications

by

Laura C. A. Rosella

H.B.Sc., University of Toronto, 2003

M.H.Sc., University of Toronto, 2005

A Thesis submitted in conformity with the requirements of the degree of:

Doctor of Philosophy (PhD) in Epidemiology

DEPARTMENT OF PUBLIC HEALTH SCIENCES

DALLA LANA SCHOOL OF PUBLIC HEALTH

UNIVERSITY OF TORONTO

2009

© by Laura C. Rosella 2009

A population based approach to diabetes mellitus risk prediction: Methodological
advances and practical applications

Laura C.A. Rosella

Doctor of Philosophy (PhD) 2009
Dalla Lana School of Public Health, University of Toronto.

Abstract

Since the publication of the Framingham algorithm for heart disease, tools that predict disease

risk have been increasingly integrated into standards of practice. The utility of algorithms at the

population level can serve several purposes in health care decision-making and planning. A

population-based risk prediction tool for Diabetes Mellitus (DM) can be particularly valuable for

public health given the significant burden of diabetes and its projected increase in the coming

years. This thesis addresses various aspects related to diabetes risk in addition to incorporating

methodologies that advance the practice of epidemiology. The goal of this thesis is to

demonstrate and inform the methods of population-based diabetes risk prediction. This is

studied in three components: (I) development and validation of a diabetes population risk tool,

(II) measurement and (III) obesity risk. Analytic methods used include prediction survival

modeling, simulation, and multilevel growth modeling. Several types of data were analyzed

including population healthy survey, health administrative, simulation and longitudinal data.

The results from this thesis reveal several important findings relevant to diabetes, obesity,

population-based risk prediction, and measurement in the population setting. In this thesis a

model (Diabetes Population Risk Tool or DPoRT) to predict 10-year risk for diabetes, which can

be applied using commonly-collected national survey data was developed and validated.

Conclusions drawn from the measurement analysis can inform research on the influence of

measurement properties (error and type) on modeling and statistical prediction. Furthermore, the

ii
use of new modeling strategies to model change of body mass index (BMI) over time both

enhance our understanding of obesity and diabetes risk and demonstrate an important

methodology for future epidemiological studies. Epidemiologists are in need of innovative and

accessible tools to assess population risk making these types of risk algorithms an important

scientific advance. Population-based prediction models can be used to improve health planning,

explore the impact of prevention strategies, and enhance our understanding of the distribution of

diabetes in the population. This work can be extended to future studies which develop tools for

disease planning at the population level in Canada and to enrich the epidemiologic literature on

modeling strategies.

iii
Supervisors: Dr. Douglas Manuel/ Dr. Cam Mustard,
Dalla Lana School of Public Health, University of Toronto

Committee Members:
Dr. Paul Corey, Dept. of Public Health Sciences, University of Toronto
Dr. Jan Hux, Health Policy Management and Evaluatio, University of Toronto
Dr. Les Roos, Dept. of Community Health Sciences, University of Manitoba
Dr. Thérèse A. Stukel, Dept. of Health Policy Management and Evaluation, University of
Toronto

Acknowledgements

I would like to thank my supervisors and committee for their brilliant mentorship and guidance.
Each has provided a unique contribution to my training and this work.

I would like to acknowledge the valuable insight of my classmates: Jennifer Bethel, Sarah Jane
Taleski and Marcelo Urquia and my colleagues at ICES: Nick Daneman, Jeff Kwong, and Refik
Saskin.

I would like to thank my husband Luigi for his unending patience, support and love. Also, I
would like to thank my parents, and my older brothers and sisters who have encouraged and
inspired me. I am blessed to have all their support and without them this work would not be
possible.

iv
Table of Contents
1 THESIS INTRODUCTION ................................................................................................................1
1.1 Introduction/Background ....................................................................................................1
1.2 References .........................................................................................................................10
2. DEVELOPMENT AND VALIDATION OF DPORT .........................................................................16
2.1 Abstract .............................................................................................................................16
2.2 Introduction .......................................................................................................................18
2.3 Methods .............................................................................................................................21
2.4 Results ...............................................................................................................................29
2.5 Discussion .........................................................................................................................40
2.6 References .........................................................................................................................46
3. MEASUREMENT IN RISK PREDICTION MODELS .......................................................................51
3.1 The influence of measurement error on accuracy (calibration), discrimination, and
overall estimation of a risk prediction model ........................................................................51
3.1.2 Abstract .....................................................................................................................51
3.1.2 Introduction ...............................................................................................................53
3.1.3 Methods .....................................................................................................................57
3.1.4 Results ........................................................................................................................66
3.1.5 Discussion ..................................................................................................................81
3.1.6 References ..................................................................................................................87
3.2 The role of ethnicity in the population-based prediction of diabetes ............................90
3.2.2 Abstract .....................................................................................................................90
3.2.2 Introduction ...............................................................................................................92
3.2.3 Methods .....................................................................................................................95
3.2.4 Results ......................................................................................................................105
3.2.5 Discussion ................................................................................................................115
3.2.6 References ................................................................................................................121
4. OBESITY RISK ..........................................................................................................................126
4.1 Abstract ...........................................................................................................................126
4.2 Introduction .....................................................................................................................128

v
 4.3 Methods ...........................................................................................................................130
4.4 Results .............................................................................................................................139
4.5 Discussion .......................................................................................................................157
4.6 References .......................................................................................................................163
5. CONCLUSION ............................................................................................................................167
6. APPENDIX .................................................................................................................................172
6.1 Glossary of frequently used terms and acronyms ...........................................................173
6.2 Ethics ..............................................................................................................................176
6.3 Study flow diagram and SAS code for simulation ..........................................................177
6.4 An example of ICC in the context of BMI distribution in the population ......................184
6.5 Detailed results from simulations ....................................................................................187

vi
1. Thesis Introduction

1.1 Introduction/Background

In many scientific disciplines, studies that predict or forecast what will happen in the future have

contributed to our understanding of the world. The value of scientific studies that provide

models to inform strategies that can modify and possibly mitigate future events is of importance

to society. Examples include estimating the impact of climate or environmental changes on the

earth‘s ecosystems or the impact of policy changes on the economy (1-3). These prediction

models have been accepted as valuable tools by scientists and have provided critical information

for the development of strategies to modify predicted trends (4, 5).

In the field of epidemiology, prediction models are underrepresented and the concept of

risk prediction is overshadowed by the estimation of relative risk measures to clarify etiological

perspectives of disease. Etiological models use the same estimation procedures as most

predictive modeling (i.e., regression) in order to quantify the relative risk associated with a

particular exposure on an outcome. Though regression is often used for both purposes, the way

in which the model is constructed will differ due to the goals of the model. The goal of a

multivariate etiological model is to optimize the accuracy of the relative risk estimate by

controlling for confounding. The goal of a prediction model differs in several important ways.

First, the outcome which needs to be optimized is an absolute measure of risk, often expressed as

percentage or probability (versus a risk or hazard ratio). Second, the goal of a prediction model is

to maximize the ability to discriminate between at risk groups and to correctly classify true risk,

known as discrimination and calibration. Typically these indices are not evaluated in etiological

models. Thirdly, prediction models must be generalizable in other populations to which the

model can be applied. Typically etiological models fit the data used to generate the relative risk

1
estimate as tightly as possible and as a result may not be reproducible using data in other settings

or may not be applicable to describe risk in another population. These goals change the criteria

for model assessment and concomitantly the methodological framework being used.

In medicine, prediction models, in the form of risk algorithms, have been used as tools for

patient decision-making. A risk algorithm is a tool used to estimate the absolute risk of an

outcome for an individual as a function of their baseline characteristics. Typically, risk is

expressed as the probability of dying or developing a disease in a given time period (6). In

medicine, risk algorithms have contributed important advances in individual patient treatment

and disease prevention. One of the most utilized risk tools is the Framingham Heart Score (7).

This tool is used to calculate the probability that a patient will develop coronary heart disease in

5 or 10 years, and has been widely integrated into cardiovascular disease prevention and

management throughout the world (8-12). Risk algorithms are widely recommended by medical

societies for appropriate identification of patients that will benefit from specific interventions.

This is exemplified in clinical guidelines for pharmacologic interventions such as cholesterol-

lowering medications (13).

Typically, risk prediction tools are used clinically and applied at the individual level.

Several potential benefits may be realized by extending the application of these tools to the

population level. Similar to the individual level, at the population setting predictive risk tools

have the potential of providing insight into the future burden of a disease in an entire region or

nation and the influence of specific risk factors. These tools can support health care decision-

making, including the effective and efficient allocation and distribution of health care resources

and planning for effective disease prevention interventions. To date, prediction tools specifically

designed for use at the population level are neither created nor used for planning.

2
Epidemiology of Diabetes

Diabetes is a chronic endocrine disorder affecting the body‘s metabolism and resulting in

structural changes affecting the organs of the vascular system. Serious complications resulting

from diabetes include coronary heart disease, stroke, retinopathy, renal failure, peripheral artery

disease, and neuropathy (14). The two main forms of diabetes are type I diabetes and type 2

diabetes. Type 1 diabetes is a result of pancreatic islet beta-cell destruction usually due to an

autoimmune response which results in insulin deficiency requiring exogenous insulin to prevent

serious complications (15). Type 2 diabetes is characterized by insulin resistance and/or

abnormal insulin secretion(15). In people with type 2 diabetes, blood sugar must be controlled

either through diet, with oral hypoglycemic drugs or in severe cases with exogenous insulin (16,

17). Type 2 diabetes accounts for over 90% of all diabetes cases worldwide (18) and is the focus

of this thesis dissertation.

Diabetes and its complications is a leading cause of death and disability, reducing overall

life expectancy and healthy life expectancy. Global estimates place the number of people with

diabetes at approximately 200 million and increasing rapidly (19). Over 2 million Canadians

have diabetes and prevalence is increasing annually (20). In Canada‘s largest province, Ontario,

the 2005 prevalence estimate of diabetes had already exceeded the rate that was predicted by the

World Health Organization (WHO) for 2030 (21). There is a growing concern that if left

unchecked, these trends may slow or even reverse life expectancy gains in the US and other

developed countries (18). In Ontario alone, diabetes has been shown to reduce healthy life

expectancy by 2.7 and 2.9 years respectively (22). The economic and medical consequences of

3
complications arising from diabetes, including cardiovascular disease and kidney failure, are

significant. Diabetes and its associated complications pose a significant economic burden to the

health care system in Canada, which was estimated at 1.6 billion in 1998 (23).

Excess body weight or obesity is associated with insulin resistance (24) and is

overwhelmingly associated with incidence of type 2 diabetes (25-29). Furthermore, it has now

been demonstrated through randomized trials that diabetes can be prevented or delayed through

reduction in body weight (30-33) and consequently intervention strategies that target weight

reduction or prevention of weight gain are largely recognized as integral to primary prevention

of diabetes (34, 35). Currently, more than 1.1 billion adults are overweight worldwide including

312 million who are obese (36). In Canada, obesity doubled between 1985 and 1998 (37) and

currently approximately 6.8 million adults ages 20 to 64 are overweight (including 4.5 million

who are obese) (38). Therefore, obesity in Canada is a significant problem (39) that must be

considered for population planning and health intervention for diabetes.

In addition to obesity several lifestyle, environmental and demographic factors have been

associated with diabetes, the main ones being: ethnicity, physical activity, alcohol and tobacco

(40). There is growing evidence that certain ethnic groups are at increased risk for developing

type 2 diabetes. Globally, non-European populations have a higher proportional burden of type 2

diabetes compared to the other regions of the world(41). The highest diabetes rates in the world

are seen in aboriginal population, including those in Australia(42, 43), United States(44, 45), and

Canada(46, 47). In the United States, studies have shown that those of African and Hispanic

decent are at increased risk for developing diabetes compared with non-Hispanic white

Americans (48-50). Throughout the world, those of South Asian decent have been shown to

carry an increase burden of type 2 diabetes compared with both non-white and white ethnicities

4
(51-53). The variability in risk between ethnicities may be due to a contribution of genetic

factors, which can influence insulin metabolism and predisposition to weight gain(54, 55). On

the other hand, the clustering of risk factors for type 2 diabetes within both families and cultural

groups may partly explain the variability in risk (56-59). The debate on the relative contributions

of genetic susceptibility versus environmental and lifestyle factors is ongoing and will continue

to be discussed as more scientific evidence emerges (60).

Both intervention and observational studies have supported the reduced incidence of type

2 diabetes associated with increased physical activity (61-71). Physical activity can reduce

diabetes risk by eliciting physiological changes at the metabolic level related to insulin-

stimulating pathways (72) in addition to aiding in the maintenance of a healthy body weight.

Several observational studies have now confirmed that shown that moderate alcohol

consumption is associated with a reduced risk of developing diabetes (73-77) whereas smoking

appears to increase risk of diabetes (78-80).

Population health planning for diabetes

Estimating future burden of diabetes is a valuable aspect to population health planning,

especially given the current high prevalence of diabetes, the startling rise in the occurrence of

obesity, and the substantial associated costs and consequences of the disease for the health

system and individuals. Population estimates of future diabetes burden can be found in the

literature using a variety of different techniques. Previous studies that estimate future diabetes

burden have either extrapolated overall trends in diabetes prevalence or indirectly incorporated

information on the influence of risk factors with various assumptions (19, 41, 48, 81, 82).

Previous studies of diabetes lifetime risk and life expectancy are not predictive, rather they

5
describe diabetes from a life-course perspective using a period or stationary population approach

(22, 48). Clinical risk algorithms have been developed for use in a clinical primary care setting.

These types of algorithms are limited in their ability to be used at the population level because

they are developed for the individual (versus the population) and require clinical data which is

infrequently available at the population level such as fasting blood sugar (83-85), or require

detailed information such as diabetes family history (86, 87).

Population planning tools for diabetes in the form of a population-based risk algorithm

has not yet been developed. Estimates of future incidence of diabetes using a risk algorithm

approach and based on current baseline risk factors in the population will alert policy makers,

planners, and physicians to the extent of the diabetes epidemic in communities. In addition, the

effectiveness of widespread prevention strategies can be improved by knowing which groups to

target and how extensive a strategy is needed to stabilize or reduce the number of new cases. In

order to maximize the benefits of a population-based risk prediction tool for diabetes, it

necessarily must be available to the widest audience and tightly integrated into existing risk

factor surveillance systems to maximize efficiency. Therefore, an effective tool is required that is

valid statistically but also meaningful and accessible in Canadian communities.

Thesis Objectives

The goal of this thesis is to demonstrate and inform the methods of population-based diabetes

risk prediction. This goal will be investigated in three broad areas: (i) development and

validation of a population-based risk tool; (ii) measurement; and, (iii) modeling of obesity trends.

These components will be investigated using different types of data and distinct analytic

approaches.

6
The first objective of this thesis is to develop and validate a risk tool to predict the probability of

developing diabetes in the subsequent 10 years (10-year risk) for Canadians using widely

available data. Thos model developed in this study is known as the Diabetes Population Risk

Tool (or DPoRT). This is the first time that a risk tool has been developed specifically for the

purpose of population health assessment and using regularly collected national data. The

conceptual framework of this model can be extended to future studies to develop tools for

disease planning at the population level in Canada and to enrich the epidemiologic field on

predictive modeling strategies. Epidemiologists are in need of innovative and accessible tools to

assess population risk thereby making these risk algorithms an important scientific advance. The

development and application of this novel population-based risk tool for diabetes raises

important questions concerning the measurement of risk factors in the population and

understanding obesity trends in the population.

The second objective is to understand the influence of measurement in risk prediction.

Indeed, there are few issues more important to epidemiology than measurement. Though

significant research is devoted to this area for relative risk estimation, there exists little

information on how measurement error and types of variables influence a prediction algorithm.

Measurement in the context of a prediction algorithm can differ from measurement in the context

of an etiological model in several important ways. Firstly, the influence of measurement

properties on risk prediction may differ from the influence on relative risk estimation in

etiological models because they rely on different endpoints for their assessment. Secondly, the

data needed for population prediction must be generalizable to the population in addition to

being regularly collected and reported on and this has implications for the type of data that can

be used to make the predictions. Specifically the scale of this type of data may require reliance

7
on methods (such as self-reporting), which may be more prone to error. In addition, the level of

detail available at the population level may be more limited than clinical data used in

epidemiological studies. The first objective in the measurement section is to use simulation to

understand the impact of measurement error on risk algorithm performance. Simulation is a

useful tool for epidemiologists which allow one to carry out investigations which otherwise may

not be possible due to lack of data. Specifically this section will quantify and describe the effect

measurement error (differential and non-differential) in self-reported height and weight on the

performance (discrimination and accuracy) and outcome (predicted risk) of a diabetes risk

algorithm. The second objective in this section is to investigate the influence of scope of

variables (or lack thereof) available at the population level. Detailed ethnic information is

collected but not publicly reported in Canada. Therefore in keeping with the goal of DPoRT to

use publicly available population health surveillance data, DPoRT uses the form of ethnicity

which is publicly reported as ―white/non-white‖. The purpose of this section is to assess the

impact of this data restriction by comparing DPoRT to an algorithm for diabetes that uses

detailed ethnicity and to report on its relative importance in terms of improvement to the

predictive accuracy of the model and policy implications. This work also provides insight into

the role of ethnicity on diabetes risk in the context of other risk factors.

The third segment of this thesis is focused on studying obesity trends over time using

longitudinal data and multilevel growth modeling. The objective of this section is to study how

lifestyle and socio-demographic factors are associated with weight gain, which is key for

preventing diabetes since obesity is its most influential risk factor. The results from such an

analysis are useful for the Diabetes Population Risk Tool for two reasons. First it can provide

further insight into the likely trajectories of obesity over time and this can be used to improve

8
DPoRT predictions or to allow for predictions farther into the future. Secondly, the results of the

model can be used to determine characteristics of the population that can be used to identify

those who are most likely at risk for weight gain (and thus are increased risk for developing

diabetes). In addition to elucidating the factors that modify weight gain, the multilevel growth

model can identify factors that influence the rate at which it occurs over time. This can be used

better inform interventions that are assessed using DPoRT.

This thesis is an investigation into diabetes, obesity, population-based risk prediction,

analytic methods, and measurement in the population setting. This thesis describes the

development and validation of DPoRT, which can be applied using commonly-collected national

survey data. Conclusions drawn from the measurement analysis will inform research on the

influence of measurement properties (error and type) on modeling and statistical prediction.

Furthermore, the use of novel modeling strategies to model change of BMI provides important

information on factors that influence the relationship between obesity and diabetes and

demonstrates an important methodology for future epidemiological studies.

9
1.2 References

1. Stern N et al. Stern Review: The Economics of Climate Change. London: HM

Treasury, 2006.
2. Hulme M et al. Relative impacts of human-induced climate change and natural
climate variability. Nature 1999;397:688-91.
3. Steigerwald DG, Stuart C. Econometric estimation of foresight: Tax policy and
investment in the United States. Review of Economics and Statistics 1997;79:32-40.
4. Ayres I. Supercrunchers: Why thinking-by-numbers is the new way to be smart.
Random House, 2007.
5. Orrell D. Apollo's Arrow: The Science of Prediction and the Future of
Everything. Toronto: HarperCollins Publishers, 2006.
6. Diamond GA. What price perfection? Calibration and discrimination of clinical
prediction models. Journal of Clinical Epidemiology 1992;45:85-9.
7. Anderson KM et al. An Updated Coronary Risk Profile - A Statement for Health-
Professionals. Circulation 1991;83:356-62.
8. Hense HW et al. Framingham risk function overestimates risk of coronary heart
disease in men and women from Germany-results from the MONICA Augsburg and the
PROCAM cohorts. European Heart Journal 2003;24:937-45.
9. Kannel WB. Some Lessons in Cardiovascular Epidemiology from Framingham.
American Journal of Cardiology 1976;37:269-82.
10. Leaverton PE et al. Representativeness of the Framingham risk model for
coronary heart disease mortality: A comparison with a national cohort study. Journal of
Chronic Diseases 1987;40:775-84.
11. Thomsen TF et al. A cross-validation of risk scores for coronary heart disease
mortality based on data from the Glostrup Population Studies and Framingham Heart
Study. International Journal of Epidemiology 2002;31:817-22.
12. Wilson PWF et al. Prediction of coronary heart disease using risk factor
categories. Circulation 1998;97:1837-47.
13. Manuel DG et al. Effectiveness and efficiency of different guidelines on statin
treatment for preventing deaths from coronary heart disease: modeling study. BMJ
2005;332:1419-22.
14. Nathan DM. Long-Term Complications of Diabetes-Mellitus. New England
Journal of Medicine 1993;328:1676-85.

10
15. Alberti KGMM, Zimmet PZ. Definition, diagnosis and classification of diabetes
mellitus and its complications. Part 1: Definition, diagnosis and classification of diabetes
mellitus provisional report of a WHO consultation. Diabetic Medicine 1998;15:559-3.
16. Gannon MC, Nuttall FQ. Control of blood glucose in type 2 diabetes without
weight loss by modification of diet composition. Nutrition and Metabolsim 2006;3.
17. Inzicchi SE. Oral antiglycemic therapy for type 2 diabetes - scientific review.
JAMA: The Journal of the American Medical Association 2002;287:360-72.
18. Zimmet P, Alberti KGMM, Shaw J. Global and societal implications of the
diabetes epidemic. Nature 2001;414:782-7.
19. Wild S et al. Global prevalence of diabetes - Estimates for the year 2000 and
projections for 2030. Diabetes Care 2004;27:1047-53.
20. Health Canada. Diabetes in Canada - Second Edition. H49-121. 2002. Ottawa,
Health Canada. 1999.
21. Lipscombe LL, Hux JE. Trends in diabetes prevalence, incidence, and mortality
in Ontario, Canada 1995-2005: a population-based study. Lancet 2007;369:750-6.
22. Manuel D, Schultz S. Health-related quality of life and health-adjusted life
expectancy of people with diabetes in Ontario, Canada, 1996-1997. Diabetes Care
2004;27:407-14.
23. Health Canada. Economic Burden of Illness in Canada. 1998. Ottawa, Public
Health Association of Canada.
24. Kahn SE, Hull RL, Utzschneider KM. Mechanisms linking obesity to insulin
resistance and type 2 diabetes. Nature 2006;444:840-6.
25. Colditz G et al. Weight as a risk factor for clinical diabetes in women. American
Journal of Epidemiology 1990;132:501-13.
26. Colditz G et al. Weight gain as a risk factor for clinical diabetes mellitus in
women. Annals of Internal Medicine 1995;122:481-6.
27. Perry IJ et al. Prospective study of risk factors for development of non-insulin
dependent diabetes in middle aged British men. British Medical Journal 1995;310:555-9.
28. Vanderpump MPJ et al. The incidence of diabetes mellitus in an English
community: a 20-year follow-up of the Wickham Survey. Diabetic Medicine
1996;13:741-7.
29. Wilson P et al. Prediction of incident diabetes mellitus in middle-aged adults.
Archives of Internal Medicine 2007;167:1068-74.
30. Knowler WC et al. Reduction in the incidence of type 2 diabetes with lifestyle
intervention or metformin. New England Journal of Medicine 2002;346:393-403.

11
31. Li G et al. The long-term effect of lifestyle interventions to prevent diabetes in
China Da Qing Diabetes Prevention Study: a 20-year follow-up study. Lancet
2008;371:1783-9.
32. Lindstrom J et al. Sustained reduction in the incidence of type 2 diabetes by
lifestyle intervention: follow-up of the Finnish Diabetes Prevention Study. Lancet
2006;368:1673-9.
33. Tuomilehto J et al. Prevention of type 2 diabetes mellitus by changes in lifestyle
among subjects with impaired glucose tolerance. New England Journal of Medicine
2001;344:1343-50.
34. Avenell A et al. Systematic review of the long-term effects and economic
consequences of treatments for obesity and implications for health improvement. Health
Technology Assessment 2004;8:1-+.
35. Mayor S. International Diabetes Federation consensus on prevention of type 2
diabetes. International Journal of Clinical Practice 2007;61:1773-5.
36. Haslam DW, James WP. Obesity. Lancet 2005;366:1197-209.
37. Katzmarzyk PT. The Canadian obesity epidemic. Obesity Research 2002;10.
38. Statistics Canada. Canadian Community Health Survey, 2005. 2005.
39. Katzmarzyk PT. The Canadian obesity epidemic. CMAJ 2002;166:1039-40.
40. van Dam RM. The epidemiology of lifestyle and risk for type 2 diabetes.
European Journal of Epidemiology 2003;18:1115-25.
41. World Health Organization. Report of a WHO consultation on obesity, Obesity:
preventing and managing the global epidemic. 1998. Geneva, World Health
Organization.
42. Odea K. Westernization, Insulin Resistance and Diabetes in Australian
Aborigines. Medical Journal of Australia 1991;155:258-64.
43. Odea K et al. Obesity, Diabetes, and Hyperlipidemia in A Central Australian
Aboriginal Community with A Long History of Acculturation. Diabetes Care
1993;16:1004-10.
44. Mokdad AH et al. The continuing epidemics of obesity and Diabetes in the United
States. JAMA: The Journal of the American Medical Association 2001;286:1195-200.
45. Pavkov ME et al. Changing patterns of type 2 diabetes incidence among Pima
Indians. Diabetes Care 2007;30:1758-63.
46. Harris SB et al. The prevalence of NIDDM and associated risk factors in native
Canadians. Diabetes Care 1997;20:185-7.

12
47. Young TK et al. Type 2 diabetes mellitus in Canada's First Nations: status of an
epidemic in progress. Canadian Medical Association Journal 2000;163:561-6.
48. Narayan KMV et al. Lifetime risk for diabetes mellitus in the United States.
JAMA 2003;290:1884-90.
49. Winkleby MA et al. Ethnic and socioeconomic differences in cardiovascular
disease risk factors - Findings for women from the third national health and nutrition
examination survey, 1988-1994. Jama-Journal of the American Medical Association
1998;280:356-62.
50. Brancati FL et al. Diabetes mellitus, race, and socioeconomic status - A
population based study. Annals of Epidemiology 1996;6:67-73.
51. Abate N, Chandalia M. Ethnicity and type 2 diabetes - Focus on Asian Indians.
Journal of Diabetes and Its Complications 2001;15:320-7.
52. Ramachandran A et al. Risk of noninsulin dependent diabetes mellitus conferred
by obesity and central adiposity in different ethnic groups: A comparative analysis
between Asian Indians, Mexican Americans and Whites. Diabetes Research and Clinical
Practice 1997;36:121-5.
53. Ramachandran A et al. Rising prevalence of NIDDM in an urban population in
India. Diabetologia 1997;40:232-7.
54. Qi L, Hu FB, Hu G. Genes, environment, and interactions in prevention of type 2
diabetes: A focus on physical activity and lifestyle changes. Current Molecular Medicine
2008;8:519-32.
55. Qi L. Genetic effects, gene-lifestyle interactions, and type 2 diabetes. Central
European Journal of Medicine 2008;3:1-7.
56. Carmelli D, Cardon LR, Fabsitz R. Clustering of Hypertension, Diabetes, and
Obesity in Adult Male Twins - Same Genes Or Same Environments. American Journal of
Human Genetics 1994;55:566-73.
57. DeFronzo RA, Ferrannini E. Insulin resistance. A multifaceted syndrome
responsible for NIDDM, obesity, hypertension, dyslipidemia, and atherosclerotic
cardiovascular disease. Diabetes Care 1991;41:173-94.
58. Martin BC et al. Familial clustering of insulin sensitivity. Diabetes 1992;41:850-
4.
59. Meigs JB et al. Risk variable clustering in the insulin resistance syndrome - The
Framingham Offspring Study. Diabetes 1997;46:1594-600.
60. Zimmet P, Alberti KGMM, Shaw J. Global and societal implications of the
diabetes epidemic. Nature 2001;414:782-7.

13
61. Bassuk SS, Manson JE. Epidemiological evidence for the role of physical activity
in reducing risk of type 2 diabetes and cardiovascular disease. Journal of Applied
Physiology 2005;99:1193-204.
62. Gill JMR, Cooper AR. Physical Activity and Prevention of Type 2 Diabetes
Mellitus. Sports Medicine 2008;38:807-24.
63. Haapanen N et al. Association between leisure time physical activity and 10-year
body mass change among working-aged men and women. International Journal of
Obesity 1997;21:288-96.
64. Helmrich SP et al. Physical activity and reduced occurrence of non-insulin-
dependent diabetes mellitus. New England Journal of Medicine 1991;325:147-52.
65. Hu FB et al. Walking Compared With Vigorous Physical Activity and Risk of
Type 2 Diabetes in Women . JAMA 1999;282:1433-9.
66. Hu G et al. Physical activity, body risk of type 2 diabetes in patients with normal
or impaired glucose regulation. Archives of Internal Medicine 2004;164:892-6.
67. Meisinger C et al. Leisure time physical activity and the risk of type 2 diabetes in
men and women from the general population. Diabetologia 2005;48:27-34.
68. Norman A et al. Total physical activity in relation to age, body mass, health and
other factors in a cohort of Swedish men. International Journal of Obesity 2002;26:670-5.
69. Petersen L, Schnohr P, Sorensen TIA. Longitudinal study of the long-term
relation between physical activity and obesity in adults. International Journal of Obesity
2004;28:105-12.
70. Villegas R et al. Physical activity and the incidence of type 2 diabetes in the
Shanghai women's health study. International Journal of Epidemiology 2006;35:1553-62.
71. Manson JE et al. Physical-Activity and Incidence of Non-Insulin-Dependent
Diabetes-Mellitus in Women. Lancet 1991;338:774-8.
72. Perseghin G et al. Increased glucose transport-phosphorylation and muscle
glycogen synthesis after exercise training in insulin-resistant subjects. New England
Journal of Medicine 1996;335:1357-62.
73. Davies MJ et al. Effects of moderate alcohol intake on fasting insulin and glucose
concentrations and insulin sensitivity in postmenopausal women - A randomized
controlled trial. Jama-Journal of the American Medical Association 2002;287:2559-62.
74. Friedman GD, Klatsky AL. Is alcohol good for your health? New England Journal
of Medicine 1993;329:1882-3.
75. Howard AA, Arnsten JH, Gourevitch MN. Effect of alcohol consumption on
diabetes mellitus - A systematic review. Annals of Internal Medicine 2004;140:211-9.

14
76. Marques-Vidal P et al. Relationships between alcoholic beverages and
cardiovascular risk factor levels in middle-aged men, the PRIME study. Atherosclerosis
2001;157:431-40.
77. Wei M et al. Alcohol intake and incidence of type 2 diabetes in men. Diabetes
Care 2000;23:18-22.
78. Manson J.E. et al. A prospective study of cigarette smoking and the incidence of
diabetes mellitus among us male physicians . American Journal of Medicine
2000;109:538-42.
79. Nakanishi N et al. Cigarette smoking and risk for impaired fasting glucose and
type 2 diabetes in middle-aged Japanese men. Annals of Internal Medicine 2000;133:183-
91.
80. Wannamethee SG, Shaper AG, Perry IJ. Smoking as a modifiable risk factor for
type 2 diabetes in middle-aged men. Diabetes Care 2001;24:1590-5.
81. Boyle JP et al. Projection of diabetes burden through 2050 - Impact of changing
demography and disease prevalence in the US. Diabetes Care 2001;24:1936-40.
82. King H, Aubert RE, Herman WH. Global burden of diabetes, 1995-2025 -
Prevalence, numerical estimates, and projections. Diabetes Care 1998;21:1414-31.
83. Eddy DM, Schlessinger L. Validation of the archimedes diabetes model. Diabetes
Care 2003;26:3102-10.
84. Hanley AJG et al. Prediction of type 2 diabetes using simple measures of insulin
resistance - Combined results from the San Antonio Heart Study, the Mexico City
Diabetes Study, and the Insulin Resistance Atherosclerosis Study. Diabetes 2003;52:463-
9.
85. Ito C et al. Prediction of diabetes mellitus (NIDDM). Diabetes Research and
Clinical Practice 1996;34:S7-S11.
86. Herman WH et al. A new and simple questionnaire to identify people at increased
risk for undiagnosed diabetes. Diabetes Care 1995;18:382-7.
87. Lindstrom J, Tuomilehto J. The diabetes risk score: a practical tool to predict type
2 diabetes risk. Diabetes Care 2007;26:725-31.

15
2. A Population Based Prediction Algorithm for the Development of Physician Diagnosed
Diabetes Mellitus: Development and Validation of the Diabetes Population Risk Tool
(DPoRT)

2.1 Abstract

Background: Diabetes prevalence is increasing in most countries and its burden poses a serious

threat to public health. National estimates of the magnitude of the upcoming diabetes epidemic

are needed to understand the distribution of diabetes risk in the population and inform health

policy and resource planning.

Objective: The objective of this study is to create and validate a population based risk prediction

tool for incident diabetes mellitus (DM) using widely available public data.

Methods: Using a cohort design that links baseline risk factors to a validated population-based

diabetes registry, a model (Diabetes Population Risk Tool or DPoRT) to predict 9-year risk for

diabetes using commonly-collected national survey data was developed and validated. The

development cohort was the Ontario‘s 1996/7 National Population Health Survey (NPHS) linked

to the validated Ontario Diabetes Database (ODD), a provincial component of the National

Diabetes Surveillance System (NDSS) in Canada. Variables were restricted to factors routinely

measured in the population. The probability of developing diabetes was modeled using sex-

specific Weibull survival functions for those > 20 years, without diabetes and not pregnant at

baseline (N = 19,861). The model was validated in two external validation cohorts: the 2000/1

Canadian Community Health Survey (CCHS) in Ontario (N=26,465) and 1996/7 NPHS in

Manitoba (N= 9,899), both linked to administrative data for NDSS-defined physician diagnosed

diabetes. Predictive accuracy was assessed by comparing observed physician diagnosed diabetes

rates with predicted risk estimates from DPoRT. Discrimination of the model was assessed

16
using a C statistic and calibration was assessed with the Hosmer-Lemeshow chi-square statistic

(χ2H-L).

Results. In the development cohort, 9-year age-standardized diabetes rates were 6.7% for males

and 5.6% for females. Predictive factors included in the final model were: BMI, age, ethnicity,

hypertension, immigrant status, smoking, education status and heart disease. The DPoRT-

predicted incidence closely agreed with the observed diabetes incidence rates in the validation

cohorts. Calibration measured using the Hosmer-Lemeshow was satisfied (χ2H-L <20) and overall

observed and predicted differed by less than 0.4%. Good discrimination was found in both

validation cohorts. In CCHS Ontario C= 0.78 95% CI (0.76, 0.80) for males and C= 0.77 95% CI

(0.75, 0.79) for females. In NPHS-Manitoba C = 0.81 95% CI (0.78, 0.83) for males and C= 0.81

95% CI (0.77, 0.83) for females.

Conclusions. The algorithm accurately predicted diabetes incidence in two validation cohorts

and demonstrated good discrimination. This algorithm can be used by health planners to estimate

diabetes incidence and quantify the impact of interventions using routinely collected survey data.

17
2.2 Introduction

In medicine, prediction tools are used to calculate risk, defined as the probability of developing a

disease or state in a given time period. Within the clinical setting, predictive studies such as the

Framingham Heart Score (1) - used to calculate the probability that a patient will develop

coronary heart disease - have contributed important advances in individual patient treatment and

disease prevention (2). Similarly, applying predictive risk tools to populations can provide

insight into the influence of risk factors, the future burden of disease in an entire region or

nation, and the value of interventions at the population level.

Global estimates place the number of people with diabetes at approximately 200 million,

and increasing rapidly (3). There is a growing concern that if left unchecked these trends may

slow or even reverse life expectancy gains in the US and other developed countries (4). Planning

for health care and public health resources can be informed by robust prediction tools. Estimates

of future diabetes incidence will alert policy makers, planners, and physicians to the extent and

urgency of the diabetes epidemic. In addition, a population prediction tool for diabetes can

identify the optimal target groups for new intervention strategies, and determine how extensive a

strategy must be to achieve the desired reduction in new cases. This insight can improve the

effectiveness and efficiency of large-scale prevention strategies.

Clinical risk algorithms have been used at the population level but with considerable

challenges (5). In particular, clinical risk tools usually require clinical data which is infrequently

available at the population level. For diabetes in particular, several clinical risk prediction tools

exist, but they require clinical data that are collected infrequently or not at all at the population

level, such as fasting blood sugar (6-8), or require detailed information such as diabetes family

history (9, 10). In addition, some prediction tools apply only to specific sub-groups of the

18
population, such as to individual within specific age ranges or only to individuals that have other

co-morbid conditions, and thus are not suitable for national population estimates (11-13).

There are several ways in which an algorithm intended for population health application

differs from an algorithm intended for clinical use. For a population algorithm the input

variables must be representative of the entire population (ideally population-based), meaningful

for health policy decisions makers, available to a wide audience, and regularly collected so that

estimates can be updated frequently. Often algorithms used in clinical settings maximized

discrimination at the expense of accuracy, meaning that the algorithms do well at rank ordering

subjects but not as well at accurately predicting actual risk (14) . In contrast, algorithms used in

populations may favor accuracy over discrimination, because population health decision makers

rely more heavily on estimates of absolute risk and numbers of disease cases versus rank-

ordering of individuals.

The objective of this study was to design and validate a diabetes risk prediction tool for

use in populations using publicly available data. The creation and application of a population-

based risk algorithm for diabetes was feasible because the risk factors for diabetes are well

known and readily measured through self-reported questionnaires in many country‘s population

health surveys. Studies have overwhelmingly shown that the most influential risk factor for the

development of diabetes is excess weight, or obesity (15, 16), which is collected nationally in

Canada. DPoRT generates diabetes risk using data that are routinely collected in Canada through

health surveys.

The objective of this study is to create a risk algorithm for diabetes incidence that can be

applied at the level of populations using widely available public data. The Diabetes Population

19
Risk Tool or DPoRT was created and validated by individually linking three different provincial

population health surveys to population-based registries of physician-diagnosed diabetes.

20
2.3 Methods

DPoRT derivation cohort

The cohort used to develop the risk algorithm was derived from 23,403 Ontario residents that

responded to the 1996/7 National Population Health Survey (NPHS-ON) conducted by Statistics

Canada that had an overall 83% response rate (17) and were linkable to health administrative

databases in Ontario. In the NPHS, households were selected though stratified, multilevel cluster

sampling of private residences using provinces and/or local planning regions as the primary

sampling unit. The survey was conducted through telephone and all responses were self-

reported. Persons under the age of 20 (n = 2, 407) and those who had self-reported diabetes were

excluded (n = 894). Those who were pregnant at the time of the survey were also excluded (n =

241), due to the fact that baseline Body Mass Index (BMI) could not be accurately ascertained,

leaving a total of 19,861 individuals (Figure 1). Sixty-six males were further excluded when

applying the algorithm due to missing baseline BMI resulting in 9,177 males and 10,618 females

in the final cohort. Respondents from the survey were individually linked to a chart-validated

registry of physician-diagnosed diabetes, the Ontario Diabetes Database, allowing each member

of the cohort to be followed to determine their diabetes status for the next 9 years (to 2005/6).

DPoRT validation cohorts

The Manitoba respondents of the 1996/7 NPHS (NPHS-MB) were used as one of two validation

cohorts. The same sampling strategy and methodology used to carry out the Ontario portion of

the NPHS was used in Manitoba. There were N = 10,118 persons linked to health care data in

Manitoba. The same exclusion criteria were applied to NPHS-MB cohort resulting in 9,899

individuals for the validation. The second validation data set was from the Ontario portion of the

21
2000/1 Canadian Community Health Survey (CCHS, Cycle 1.1, N = 37,473), a national

telephone survey administered by Statistics Canada. The target population of the CCHS

consisted of persons aged 12 and over resident in private dwellings in all provinces and

territories, excepting those living on Aboriginal reserves, on Canadian Forces Bases, or in some

remote places. The CCHS included the same self-reported health questions as the NPHS. Like

the NPHS, this survey uses a multistage stratified cluster design and provides cross-sectional

data representative of 98% of the Canadian population over the age of 12 years, and attained an

80% overall response rate (18, 19). After the exclusion criteria were applied there were 26,465

individuals in the validation cohort.

The NPHS-MB cohort had a 9-year follow-up (1996-2005) and the CCHS-ON had a 5-

year follow-up (2000-2005) therefore the predicted risks from the DPoRT algorithm were

generated for both 9-year and 5-year follow-up periods.

Figure 1. Cohort used to develop DPoRT

Females Males
N=23,403
1138 age < 20 years 1269 age < 20 years
N = 20, 996
241 pregnant at interview
N = 20, 755
452 Prior DM 442 Prior DM
N = 19, 861
No DM (N = 18, 451) DM (N = 1410)

859 females died 773 males died

692 females 718 males

N = 16, 819

22
Identifying respondents who develop diabetes

Survey data from development and validation cohorts were linked to provincial administrative

health care databases that include all persons covered under the government funded universal

health insurance plan. The diabetes status of all respondents in Ontario was established by

linking persons to the Ontario Diabetes Database (ODD). The Ontario Diabetes Database (ODD)

contains all physician diagnosed diabetes patients in Ontario identified since 1991. The database

was created using hospital discharge abstracts and physician service claims. A patient is said to

have physician diagnosed diabetes if he or she meets at least one of the following two criteria:

(a) a hospital admission with a diabetes diagnosis (International Classification of Diseases

Clinical Modification code 250 (ICD9-CM) before 2002 or ICD-10 code E10 – E14 after 2002,

or (b) a physician services claim with a diabetes diagnosis (code 250) followed within two years

by a either physician services claim or a hospital admission with a diabetes diagnosis.

Individuals entered the ODD as incident cases when they were defined as having diabetes

according to the criteria described above. A hospital record with a diagnosis of pregnancy care

or delivery close to a diabetic record (i.e. a gestational admission date between 90 days before

and 120 days after the diabetic record date), were considered to relate to a diagnosis of

gestational diabetes and therefore were excluded. The ODD has been validated against primary

care health records, and demonstrated excellent accuracy for determining incidence and

prevalence of diabetes in Ontario (sensitivity 86%, specificity of 97%)(20, 21). Information

regarding the vital statistics and eligibility for health care coverage for linked respondents was

captured from the Registered Persons Data Base (RPDB). The algorithm used to create the ODD

was applied to Manitoba‘s administrative health care data to ascertain physician-diagnosed

23
diabetes status in that province. The ODD algorithm is applied nationally using provincial

administrative registries (known as the National Diabetes Surveillance System (NDSS)) and has

been successfully validated in several Canadian provinces (22).

Variables

To ensure that DPoRT would be widely applicable across different populations, variables

considered for the algorithm had to fulfil the following criteria: (i) be based on well established

evidence, (ii) be captured in a consistent manner across surveys (i.e. using the same questions),

(iii) be unlikely to be subject to serious self-reporting error (such as alcohol and dietary habits),

and (iv) be easily captured using survey data. All potential variables were obtained from self-

report responses to NPHS and CCHS telephone surveys, including: age, height and weight,

presence of chronic conditions diagnosed by a health professional (including hypertension and

heart disease), ethnicity, immigration status, smoking status, highest level of achieved education,

total household income, alcohol consumption and physical activity (based on metabolic

equivalents). Body Mass Index (BMI) in kg/ m2 was used as an indicator of obesity. Derived

BMI, calculated by dividing the weight in kilograms by height squared (in meters squared)

directly from the NPHS, is only calculated for respondents aged 30 to 64; therefore, BMI was

calculated using weight and height values for the individual for those who fell outside the age

range of 30 – 64 (23). In order to facilitate future use in a variety of settings, all variable in the

risk algorithm were kept in the form which is released in the public use data.

24
Statistical Analysis

The goal of the model was to create a risk algorithm that would accurately predict diabetes risk

with high discrimination and calibration. Discrimination is the ability to differentiate between

those who are at high risk and those who are at low risk – or in this case those who will and will

not develop diabetes given a fixed set of variables. Calibration is the existence of close

agreement between observed diabetes risk – calculated by comparing observed diabetes rates and

predicted probabilities from DPoRT. Detailed definitions of discrimination and calibration are in

appendix 6.1. For each cohort member, the probability of physician-diagnosed diabetes was

assessed from the interview date until censoring for death or end of follow-up. The final model

was fit using a Weibull accelerated failure time model which provides simple and robust survival

probabilities. Importantly this model includes time in its equation, allowing the user to predict

probability for a range of follow-up periods. Alternative parametric models (exponential and log-

logistic) and a semi-parametric model (Cox Proportional Hazard model) were also considered

but either did not fit the data appropriately or did not perform as well, particularly when applying

to external cohorts. Furthermore, the ability to asses risk for a variety of follow-up times is an

important feature needed when applying this model in different settings.

Diabetes risk functions were derived separately for men and women above age 20

without prior diabetes diagnoses. The following covariates were considered as candidates for the

algorithm: age, BMI and their interactions, presence of hypertension, presence of heart disease,

ethnicity, immigration status, smoking status, education, income, and physical activity (based on

metabolic equivalents or METS). First unadjusted relationships with diabetes were inspected and

variables with a clinically significant unadjusted hazard ratio were considered for the final

model. Risk factors were added to the model in a nested fashion based on clinical importance,

25
and the marginal statistical and predictive significance of each group was evaluated, controlling

for variables already in the model. For both males and females, the BMI-age interactions were fit

first, then hypertension and heart disease, then ethnicity and immigration status, then remaining

risk factors. Covariates with missing information were also assessed for their relationship with

the outcome as missingness is often associated with higher risk, in this case, missing BMI for

females. Polynomial functions and splines were fit to try and accurate capture the relationship

between BMI and diabetes and its interaction with age.

Each variable was centred on the population mean before inclusion in the model to allow

for easier calibration with other cohorts. This means that when the algorithm is applied, all

variables are centered to the mean variables for the cohort to which it is being applied allowing

levels of risk to be reflective of the average baseline risk in the cohort of interest. Overall risk

(predicted probability) of diabetes for each person was calculated by multiplying the individual‘s

risk factor values by the corresponding regression coefficients, and summing the products (24).

The form of the model was assessed using likelihood ratio tests to compare nested parametric

models (25). A plot of Log(-logS(t))) vs log(t) was inspected for linearity to assess consistency

of the survival times with the Weibull distribution. Cox-Snell residual plots were also

constructed to assess the adequacy of the Weibull distribution assumption. The Cox-proportional

hazards model was fit to compare the risks with the parametric model.

Discrimination was measured using a C statistic modified for survival data developed by

Pencina et al. (26), analogous to the area under the ROC curve (27). Calibration was assessed

using a modified version of the Hosmer-Lemeshow χ2 statistic developed by Nam (28, 29). This

statistic is computed by dividing the validation cohort into deciles of predicted risk and

comparing the observed versus predicted risk in each decile using a chi-square statistic (see

26
appendix 6.1). To mark sufficient calibration χ2 = 20 was used as a cutoff (p<0.01), consistent

with D‘Agostino‘s method in validating the Framingham algorithms (28). Discrimination and

calibration statistics were also computed using the algorithm coefficients generated from the

validation cohort itself, and labeled ―own cohort‖. This was done to assess if the algorithms

generated from the validation cohort produced significantly different measures of predictive

accuracy than DPoRT. In addition to the Hosmer-Lemeshow test for calibration, graphical

representations of predicted and observed rates were produced to assess accuracy in prediction

across quintiles of risk. In order to generate these plots, predicted risks are ordered from smallest

to largest. The predicted probabilities are separated into quantiles of risk (in this study deciles or

quintiles). The number of predicted diabetes cases in each decile or quintile is determined by

summing the predicted risk within each quantile and the observed diabetes cases are also

summed within quantile. To present these numbers as proportions, the number of predicted and

observed cases are respectively divided by the number of individuals within each decile or

quintile. The observed and predicted cases are then plotted against each other at each quantile of

risk.

Re-calibration was achieved by substituting the mean values from the validation cohort to

define all variables (which were mean-centered) with the Framingham risk function. Due to

systematic case ascertainment differences between jurisdictions, a further adjustment was

applied to predicted rates outside of Ontario (to reflect case ascertainment differences between

provinces). This adjustment was done by estimating the average ratio between observed and

predicted risk across decile (after taking into account differences in risk level through mean-

centering). The aggregate risk estimate was divided by the average amount of over-prediction

averaged across all risk groups. The assumption for this adjustment was that the differences are a

27
function of case-ascertainment (due to billing practices) between provinces. This approach has

been used in previous research involving the calibration of risk algorithms, including by Brindle

et al. to adjust Framingham risk functions for the UK (30).

All estimates (including betas and variance estimates) incorporated bootstrap replicate

survey weights to accurately reflect the demographics of the Canadian population and account

for the survey sampling design based on selection probabilities and post stratification

adjustments. Variance estimates and 95% confidence intervals were calculated using bootstrap

survey weights (31, 32). All statistics were computed using SAS statistical software (version 9.1

SAS Institute Inc, Cary, NC).

28
2.4 Results

In the development cohort, 718 males and 692 females developed physician diagnosed diabetes

within the 9-year follow-up period (crude 9-year incidence rate of 7.78% and 6.13%

respectively). The age standardized (standardized to the 1991 Canadian population) 9-year

incidence rates in the development cohorts were 6.67 % for males and 5.59 % for females. The

5-year age standardized incidence rates in the development cohort were 3.59 % for males and

2.81% for females. In the Manitoba validation cohort 272 males and 258 females developed

physician diagnosed diabetes within the 9-year follow-up period (crude 9-year incidence rate of

7.22% and 4.75% respectively). The age standardized 9-year incidence rates in the development

cohorts were 6.55 % for males and 4.27 % for females. In the Ontario-CCHS validation cohort

559 males and 558 females developed physician diagnosed diabetes within the 5-year follow-up

period (crude 5-year incidence rate of 4.60% and 3.69% respectively). The age standardized

5-year incidence rates in the Ontario-CCHS validation cohort was 3.95 % for males and 3.35 %

for females. All baseline population characteristics in the derivation cohort and two validation

cohorts are shown in table 1. Both the Manitoba and Ontario validation cohorts differed from

the derivation cohort. There were similar in age distribution, however both NPHS-MB and

CCHS-ON had a higher proportion of obese individuals. The CCHS-ON cohort had significantly

higher number non-white ethnicities compared to the derivation cohort, whereas the derivation

cohort and Manitoba validation cohort were similar. Compared to the derivation cohort the

CCHS-ON had a higher baseline prevalence of hypertension and heart disease but a lower

prevalence of smoking. The Manitoba validation cohort had higher levels of hypertension and

heart disease compared to the derivation cohort in women only.

29
 Table 1. Baseline characteristics of development and validation cohorts. Categorical variables
are represented as a proportion (%) and continuous variables are represented as a mean/median.

MALES FEMALES
Development Validation Cohorts Development Validation Cohorts
Cohort Cohort
Risk Factor Ontario Manitoba Ontario Women Manitoba Ontario
NPHS NPHS CCHS NPHS NPHS CCHS
(N = 9,177) (N=4,670) (12, 020) (N = 10,618) (N=5,229) (N=14,445)
Body Mass Index 26.10/ 26.86/ 26.12/ 24.47/ 25.43/ 24.98/
(Kg/m2) 25.70 26.31 25.62 23.50 24.59 24.03
Age (mean/median) 44/42 44/42 44/42 46/42 47/42 46/42

Age <45, % 54.80 55.67 55.85 51.68 52.71 51.59

45≤Age<65, % 30.78 29.71 31.00 29.92 27.79 31.51
Age≥65, % 14.42 14.63 13.15 18.39 19.51 16.90

BMI<23 19.48 17.79 22.23 40.39 35.89 39.29

23≤BMI<25 22.11 20.34 21.51 19.01 16.65 17.79
25≤BMI<30 43.97 44.34 40.03 24.36 28.51 27.19
30≤BMI<35 11.31 14.40 12.74 8.50 10.55 9.47
BMI≥35 2.40 2.63 3.05 2.77 3.15 4.11
BMI = missing 0.73 0.51 0.44 4.98 5.26 2.14

Non-white, % 11.51 10.42 16.68 10.41 10.51 16.76

Hypertension, % 10.23 10.16 12.50 12.32 13.22 14.94
Current Smoker, % 29.67 30.76 24.78 24.48 24.40 18.97
Physical Activity -
Mets (kcal/day) 1.86/1.20 1.79/1.10 1.97/1.30 1.62/1.10 1.44/1.00 1.63/1.10
Heart Disease, % 4.97 4.43 5.19 4.16 4.62 5.24
Graduated Post
Secondary School, % 81.12 73.28 82.11 81.86 73.81 81.09
Number Incident
Diabetes (unweighted) 718 272 559 692 258 558
Crude 9-year
incidence rate 7.78 7.22 6.13 4.75
Age standardized*
9-year incidence rate 6.67 6.55 5.59 4.27
Crude 5-year
incidence rate 4.26 4.60 3.23 3.69
Age standardized*
5-year incidence rate 3.59 3.95 2.81 3.35
*standardized to the 1991 Canadian population

30
For both males and females, diabetes risk was highly positively related to BMI and age (tables

2a&b). BMI was considered in both its continuous and categorical form. It was found the

relationship between BMI and diabetes was non-linear and complicated power transformations

were needed to correctly model the association. Continuous BMI not only had a complicated

exponential form but its relationship with diabetes varied significantly by age and sex. BMI

categories had the advantage of not being constrained to a specific mathematical function and

having regression coefficients that were easier to interpret than those from a series of polynomial

BMI terms. Ultimately, the best goodness-of-fit and calibration was achieved by categorizing

BMI and including its interactions with age since the effect of BMI was highly dependent on

age. This categorization was least likely to over-fit when applied to other data while at the same

remaining discriminating.

Non-white ethnicity, hypertension, and less than post secondary education were also

important factors associated with an increased risk of diabetes in the multivariate model. For

males, smoking and heart disease were also important independent risk factors that were found to

improve model characteristics; for women, immigrant status was an additional important risk

factor found to improve the model. Some variables were excluded from both models because

they did not improve the model or worsened predictive accuracy, including: income, physical

activity, and alcohol consumption. The majority of discrimination (C=0.70) was achieved by

including only age, BMI and their interaction. The sex-specific estimates for DPoRT algorithm

are shown in table 3. Figure 2 demonstrates how the risk coefficients in DPoRT were used to

calculate risk using a high-risk male as an example.

31
Table 2a. DPoRT multivariate-adjusted hazard ratios and 95% confidence intervals for 9-year
physician diagnosed diabetes for males

Risk Factor Males

Hazard ratio 95% CI
Hypertension
No 1.00
Yes 1.38 (1.08, 1.78)*
Non-white ethnicity
No 1.00
Yes 2.19 (1.54, 3.10) ‡
Heart Disease
No 1.00
Yes 1.95 (1.43, 2.65) ‡
Current Smoker
No 1.00
Yes 1.25 (1.00, 1.57)
Education
< Post-secondary 1.00
Secondary 0.75 (0.61, 0.92)†
Age-BMI category
BMI<23*Age<45 1.00
23≤BMI<25*Age<45 4.65 (1.53, 14.12) ‡
25≤BMI<30*Age<45 6.85 (2.41, 19.46) ‡
30≤BMI<35*Age<45 23.58 (7.89, 70.43) ‡
BMI≥35*Age<45 74.68 (24.23, 230.17) ‡
BMI<23*Age≥45 11.70 (3.97, 34.50) ‡
23≤BMI<25*Age≥45 20.79 (7.27, 59.50) ‡
25≤BMI<30*Age≥45 34.44 (12.17, 97.48) ‡
30≤BMI<35*Age≥45 61.69 (20.75, 183.38) ‡
BMI≥35*Age≥45 86.04 (26.61, 278.20) ‡

Adjusted for all variables in the model simultaneously; all standard errors are computed
using survey bootstrap weights

*0.01<p<0.05, †0.001<p<0.01, ‡p<0.001

32
Table 2b. DPoRT multivariate-adjusted hazard ratios and 95% confidence intervals for 9-year
physician diagnosed diabetes for females

Risk Factor Females

Hazard ratio 95% CI
Hypertension
No 1.00
Yes 1.44 (1.11, 1.88)†
Non-white ethnicity
No 1.00
Yes 1.74 (1.16, 2.60)†
Immigrant Status
No 1.00
Yes 1.45 (1.17, 1.81)‡
Education
< Post-secondary 1.00
Secondary 0.77 (0.61, 0.97)*
Age-BMI category
BMI<23*Age<45 1.00
23≤BMI<25*Age<45 2.00 (0.89. 4.50)
25≤BMI<30*Age<45 2.94 (1.47, 5.92)†
30≤BMI<35*Age<45 6.08 (3.01, 12.28)‡
BMI≥35*Age<45 13.75 (6.27, 30.17)‡
BMI = missing*Age<45 4.26 (1.95, 9.33)‡
BMI<23*45≤Age<65 0.91 (0.41, 2.04)
23≤BMI<25*45≤Age<65 2.45 (1.18, 5.10)*
25≤BMI<30*45≤Age<65 6.13 (3.23, 11.63)‡
30≤BMI<35*45≤Age<65 17.03 (8.63, 33.60)‡
BMI≥35*45≤Age<65 18.25 (8.69, 18.33)‡
BMI = missing*45≤Age<65 9.11 (4.29, 19.31)‡
BMI<23*Age≥65 4.00 (2.05, 7.77)‡
23≤BMI<25*Age≥65 4.31 (2.12, 8.75)‡
25≤BMI<30*Age≥65 7.75 (4.16, 14.45)‡
30≤BMI<35*Age≥65 11.75 (5.43, 25.42)‡
BMI≥35*Age≥65 16.61 (6.12, 45.06)‡
BMI = missing*Age≥65 10.38 (2.75, 39.21)‡

Adjusted for all variables in the model simultaneously; all standard errors are computed
using survey bootstrap weights

*0.01<p<0.05, †0.001<p<0.01, ‡p<0.001

33
Table 3. DPoRT Functions for predicting 9-year physician diagnosed diabetes for a) males and b) females

(a) (b)
Risk Factor Men Risk Factor Women
Intercept 10.5971 Intercept 10.5474
Hypertension Hypertension
No 0.00 Yes 0.00
Yes -0.2624 No -0.2865
Non-white Ethnicity Non-white Ethnicity
No 0.00 Yes 0.00
Yes -0.6316 No -0.4309
Heart Disease Immigrant Status
No 0.00 Yes 0.00
Yes -0.5355 No -0.2930
Current Smoker Education
No 0.00 < Post-secondary 0.00
Yes -0.1765 Secondary 0.2042
Education Age-BMI category
< Post-secondary 0.00 BMI<23*Age<45 0.00
Secondary 0.2344 23≤BMI<25*Age<45 -0.5432
Age-BMI category 25≤BMI<30*Age<45 -0.8453
BMI<23*Age<45 0.00 30≤BMI<35*Age<45 -1.4104
23≤BMI<25*Age<45 -1.2378 BMI≥35*Age<45 -2.0483
25≤BMI<30*Age<45 -1.5490 BMI = missing*Age<45 -1.1328
30≤BMI<35*Age<45 -2.5437 BMI<23*45≤Age<65 0.0711
BMI≥35*Age<45 -3.4717 23≤BMI<25*45≤Age<65 -0.7011
BMI<23*Age≥45 -1.9794 25≤BMI<30*45≤Age<65 -1.4167
23≤BMI<25*Age≥45 -2.4426 30≤BMI<35*45≤Age<65 -2.2150
25≤BMI<30*Age≥45 -2.8488 BMI≥35*45≤Age<65 -2.2695
30≤BMI<35*Age≥45 -3.3179 BMI = missing*45≤Age<65 -1.7260
BMI≥35*Age≥45 -3.5857 BMI<23*Age≥65 -1.0823
Scale 0.8049 23≤BMI<25*Age≥65 -1.1419
25≤BMI<30*Age≥65 -1.5999
30≤BMI<35*Age≥65 -1.9254
BMI≥35*Age≥65 -2.1959
BMI = missing*Age≥65 -1.8284
Scale 0.7814

34
 Figure 2. Example use of DPoRT to predict the 9 year risk of a diabetes mellitus for a specific
high-risk male
2
Profile: Male; 55 years old; BMI = 29 kg/m , hypertension, white, non- smoker; heart disease & hypertension,
graduated secondary school
**Note all variables are centered

μ = 10.5971 + -0.2624*hypertension -0.6316*non-white ethnicity -0.5355*heart disease -0.1765*smoker

+0.2344*secondary school education + 0.00*BMI<23*Age<45 -1.2378* 23≤BMI<25*Age<45
1.5490*25≤BMI<30*Age<45 -2.5437*30≤BMI<35*Age<45 -3.4717*BMI≥35*Age<45 -1.9794*BMI<23*Age≥45 -
2.4426*23≤BMI<25*Age≥45 -2.8488*25≤BMI<30*Age≥45
-3.3179*30≤BMI<35*Age≥45 -3.5857* BMI≥35*Age≥45

μ = 10.5971 + -0.2624*(1 - 0.1083811) -0.6316*(0-0.108281) -0.5355*(1-0.0519083) -0.1765*(0- 0.2939450) +

0.2344*(1-6288460) - 1.2378* (0-0.1290286) - 1.5490*(0-0.2207441) -2.5437*(0-0.0558456) -3.4717*(0-0.0120304)
1.9794*(0-0.068159) -2.4426*(0- 0.0860440) - 2.8488*(0.2189925)
-3.3179*(0-0.0572891) -3.5857* (1-0.0119579)

μ = 8.8816
m = log(365.25*9) – μ / σ = 8.09781– 8.8816/ 0.8049
m = -0.97374
9-year predicted risk for developing diabetes is:
-0.97374
P = 1-exp(-e )
P = 0.314542896
or 31.45%

The algorithm was re-calibrated to the mean of the validation cohorts. For example, the

coefficient for the ‗non-white ethnicity‘ variable in males is -0.6316 (table 3a). If the variable

were not mean centered this coefficient would be multiplied by 1 or 0 given the status of

ethnicity for that individual; therefore for someone with non-white ethnicity it would be -0.6316

x 1. By mean centering the value of the individual is related to the baseline rate of non-white

ethnicity in the cohort to which the algorithm is being applied. If the cohort where the algorithm

is being applied has a 10.828% baseline rate of non-white ethnicity for males, then the resulting

arithmetic becomes: -0.6316 x (1-0.10828).

Observed and predicted diabetes rates differed ≤ 0.4% in the two validation cohorts. In

the CCHS cohort, the mean-calibrated DPoRT 5-year predicted (and observed) DM incidence
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

rates were 4.2% (4.6%) for males and 3.4% (3.7%) for females. Observed and predicted risks

for the CCHS cohorts, by decile of risk, are shown in figure 2. In the NPHS-MB cohort, the

mean-calibrated DPoRT 9-year DM predicted (observed) incidence rates were 8.2% (7.1%) for

males and 6.6% (4.7%) for females. After adjusting for case ascertainment differences between

provinces, the predicted (observed) incidence rates were 7.0% (7.1%) for males and 5.1% (4.7%)

for females (figure 2).

Figure 2. Overall calibrated predicted versus observed physician diagnosed diabetes rates for
males and females in the two validation datasets.

Observed Predicted
8.0%

7.0%

6.0%
Diabetes incidence rate (%)

5.0%

4.0%

3.0%

2.0%

1.0%

0.0%
Males Females Males Females
CCHS-ON validation cohort NPHS-MB validation cohort

36
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Due to the smaller sample size in the NPHS-MB cohort, the risks are reported by quintiles. R-

squared between observed diabetes incidence rates and predicted probabilities from the DPoRT

algorithm across deciles of risk in both validation data sets exceeded 98% for both males and

females after re-calibration. C-statistics when applying DPoRT to the validation cohorts were

high (0.77 – 0.80) and were not appreciably lower than those generated from the ―own cohort‖

models (Table 4). Hosmer-Lemeshow χ2 statistics to assess calibration are shown in table 4.

For men and women, sufficient calibration was demonstrated in the CCHS validation cohort after

mean adjustment (χ2 <20). For NPHS-MB, to achieve goodness of fit additional calibration was

needed to adjust for case ascertainment differences between provinces.

37
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 3. Calibrated predictions for incidence of physician diagnosed diabetes for males and
females in two validation datasets across deciles or quintiles of risk.

Males Females
A. DPoRT applied to Ontario CCHS †
N = 12,018 C = 0.77 95%CI (0.76, 0.79) N = 14,442 C = 0.76 95%CI (0.74, 0.77)

B. DPoRT applied to Manitoba NPHS † ¥

N = 4,670 C = 0.79 95%CI (0.77, 0.82) N = 5,229 C = 0.80 95%CI (0.77, 0.82)

† Goodness of fit was achieved (H-L χ2 < 20) calibrated model

38
 Table 4. C statistics with 95% confidence intervals and calibration χ2 statistics for DPoRT and
cohorts‘ own functions.
Men Women
NPHS ON CCHS ON NPHS MB NPHS ON CCHS ON NPHS MB
C statistic 95% CI
DPoRT 0.77 0.77 0.79 0.78 0.76 0.80
(0.76, 0.79) (0.76, 0.79) (0.77, 0.82) (0.76, 0.79) (0.74, 0.77) (0.77, 0.82)
Own Function† 0.80 0.78 0.80 0.80 0.77 0.80
(0.78, 0.83) (0.76, 0.79) (0.77, 0.82) (0.78, 0.83) (0.75, 0.79) (0.77, 0.82)
Calibration χ2
Uncalibrated DPoRT 4.33 13.23 136.13 5.22 24.84 35.07
Mean Calibrated … 13.04 32.616 … 18.27 25.83
DPoRT‡
Mean and CA Calibrated … … 18.35 … … 17.61
DPoRT¥
Own function … 8.89 8.32 … 10.44 4.88
† Own function is the factors of the algorithm applied using coefficients derived from the validation cohort‘s own
data
‡Calibrated DPoRt is function adjusted using the validation cohort‘s own means for factors; does not include case
ascertainment adjustment
¥ Based on binary H-L statistic (approximation to survival based H-L)
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

2.5 Discussion

This study demonstrated that diabetes risk can be accurately predicted at the population level

using self-reported age, sex, BMI and other measures available in population health surveys. In

addition to displaying good discrimination, DPoRT-predicted rates closely agreed with observed

rates for both males and females in both external validation cohorts, and this agreement was

generally maintained across deciles and quintiles of risk. To my knowledge, DPoRT is the first

validated risk tool that is integrated into commonly-collected population health survey data.

DPoRT offers advantages over existing methods used to estimate future diabetes risk in

populations. Previous studies that estimate future diabetes burden have either extrapolated

overall trends in diabetes prevalence or indirectly incorporated information on the influence of

risk factors with various assumptions (3, 33-36). Previous studies of diabetes lifetime risk and

life expectancy are not predictive, rather they describe diabetes from a life-course perspective

using a period or stationary population approach (36, 37). Other studies focus on overall diabetes

burden, a useful approach, but one which does not enable users to directly assess the impact of

risk factors, such as BMI, on future diabetes. Furthermore, these studies did not assess how

future diabetes can be prevented by targeting risk factors since they do not directly quantify the

influence of risk factors on baseline risk or diabetes incidence.

Complex modeling and simulation studies differ from the approach used in this study in

that they use additional information on how populations and risk factors change over time (38,

39). Other simulation studies add more detailed clinical information such as fasting blood sugar

level or information on diabetes family history, data not available at the population level. A

strength of these simulation models is that they can combine together different data sources and

study findings (40). However, these models are complex and often represent clinical or

40
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

theoretical populations, making their estimates difficult to validate in external populations that

are meaningful for population health planning. DPoRT could be incorporated into simulation

models that consider future changes in population composition and risk factors.

The nature of diabetes risk allowed us to discriminate and explain risk using a limited number of

variables – most importantly BMI. Discrimination of DPoRT (C statistic 0.77 – 0.80 and wide

range of predictive risk across deciles) is as high as or higher than many clinical risk prediction

tools used in clinical practice. The most stringent test of model accuracy is the application of the

model to a different population (41, 42). This study demonstrates that DPoRT is discriminating

and accurate in two external populations that varied across geography and time. The algorithm

was further calibrated using population means, which may attenuate differences between

populations since risk estimates are relative to baseline risk in the population.

Given current data in most countries, DPoRT is a more balanced approach to estimating

diabetes risk than methods used in previous research. A number of important clinical values are

excluded from DPoRT, such as hip to waist ratio, waist circumference, fasting blood glucose,

and family history (10;41;42). Although these variables may be clinically important for

assessing diabetes risk, adding these, or other detailed anthropometric measures, is not feasible

because they are not routinely collected in most populations. These omitted variables are

unlikely to have a major impact on the performance characteristics of the model due to the

clustering of risk factors, particularly when dealing with abnormalities of the metabolic system

(43-47). Variables not included in DPoRT, such as family history of diabetes or poor diet, are

also associated with the clustering of metabolic risk factors that are included in the algorithm

such as hypertension and BMI. There is increasing awareness that simple clinical risk tools,

including those with self-report risk factor data, perform as well or better than complex models.

41
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

(13, 43, 45-52). Adding more predictors or clinical measures results in negligible improvements

to discrimination, and this is likely due to the high correlation of these risk factors with obesity

and other variables already included in the model. Previous study have shown that additional

predictive variables need to have a high independent risk (odds ratio greater than 6.9 or greater)

to result in significant improvements in discrimination, once a discrimination of 0.8 is already

achieved (53). This phenomenon was corroborated in the creation of DPoRT, as maximal levels

of discrimination was achieved using few predictor variables.

Obesity is the most important factor in predicting diabetes risk. BMI is the most

commonly used marker of obesity; however measures of central obesity may capture the entire

risk domain in a more comprehensive manner and be more meaningful across all age groups

(54). This may be considered a limitation of DPoRT since measures of central obesity are not

routinely collected on the population level and thus are not included. If included, these estimates

may increase both discrimination and accuracy of the algorithm; however debate about the

clinical utility of these measures still continues. A recent meta-analysis has shown that there is

no evidence of difference in estimates associated with incident diabetes between BMI, waist

circumference and waist/hip ratios (55). Furthermore, algorithms to identify individuals for

weight loss in populations did not differ if using BMI or waist circumference (56).

To ensure DPoRT can be applied in different populations, we gave preference to

variables that were: based on established evidence, remained stable over time, were unlikely to

be subject to serious measurement error (such as alcohol and dietary habits), and were easily

captured using survey data in different populations. For example, physical activity has been

shown to have a protective effect on diabetes incidence (57) but was removed from the final

algorithm due to the inability to capture this in a reliable and reproducible manner across studies,

42
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

and because of it marginal improvement in the discrimination of diabetes risk in our creation

cohort. Despite placing considerable constraints on variable selection as a means of ensuring

maximum feasibility, DPoRT maintained good discrimination.

Using self-report measures is a limitation which could affect predictive risk accuracy

because these measures may be more subject to reporting error and/or bias than measured

anthropometric measures. In general there is a high correlation between self-report and measured

height and weight; however, validation studies do show that weight tends to be slightly

underestimated and height trends to be slightly overestimated and as a result reported BMI is

generally lower than measured BMI (58-60). The effect of self-reported BMI may depend on the

population where the algorithm is being applied since these patterns have been shown to vary

across gender and socioeconomic status (61-63). The ability of DPoRT‘s predictive estimates to

agree with observed diabetes risk in different populations will be reduced if systematic errors

associated with responses vary across populations or time. In chapter 3.1, simulation modeling

is used to specifically explore the impact of potential reporting biases on risk prediction.

Another limitation of this study is the use of physician-diagnosed diabetes, as opposed to

true diabetes status (diagnosed plus undiagnosed). The estimates from DPoRT may exclude

diabetics in the population who are not yet identified by a physician. This could reflect patients

with less severe disease and/or poorer access to medical care. Physician-diagnosed diabetes is

currently the most commonly measured definition of diabetes at the level of populations.

Although estimates put the true prevalence of diabetes higher, advocates of the physician-

diagnosed outcome argue that it is meaningful to people with recognized diabetes and to the

treatment of these diagnosed patients within the health care system. If diabetes testing/screening

increases over time, predicted estimates could be lower than the observed estimates (under the

43
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

assumption that this would lead to increased case detection). DPoRT has been found to be

accurate in different populations for different time periods; however, DPoRT could be re-

calibrated to predict total diabetes cases using revised information on screening/testing or using

estimates of the number of undiagnosed cases relative to diagnosed cases in the population.

Finally, the potential for inaccuracy increases the longer into the future the predictions are made

or when unforeseen changes occur; therefore, it is recommended that predictions from DPoRT

are updated frequently by using the most recent data, limiting predictive calculations to 9 years

or less, and validating the risk tool in the population where it is being applied. This tool was

developed in Canada and is likely most appropriate in the Canadian setting; however like other

risk tools, DPoRT may be transportable once validated and calibrated as needed. Careful

interpretation and adjustment for contextual variables, such as immigrant status, will need to be

made in populations outside of Canada.

Curbing the diabetes epidemic has been identified by several governments and health

policy makers as one of their top priorities for improving, and even maintaining, the health of

their nations. Population-based prediction models such as DPoRT can be used to improve health

planning, explore the impact of prevention strategies, and enhance understanding of the

distribution of disease populations. This study demonstrates that DPoRT accurately predicts

diabetes incidence and is effective at discriminating risk at the population level. This algorithm

can be used by health planners to estimate diabetes incidence, to stratify the population by risk

and quantify the impact of interventions using routinely collected survey data. Validation of

DPoRT will continue as more effective methods to quantify variation in case ascertainment in

different populations are developed. As the surveillance of risk factors and diabetes advances,

44
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

DPoRT can be adapted to become even more accurate, while maintaining its accessibility for

decision makers.

45
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

2.6 References

1. Anderson KM et al. An Updated Coronary Risk Profile - A Statement for Health-

Professionals. Circulation 1991;83:356-62.
2. Hippisley-Cox J et al. Derivation and validation of QRISK, a new cardiovascular
disease risk score for the United Kingdom: prospective open cohort study. British
Medical Journal 2007;335:136-41.
3. Wild S et al. Global prevalence of diabetes - Estimates for the year 2000 and
projections for 2030. Diabetes Care 2004;27:1047-53.
4. Zimmet P, Alberti KGMM, Shaw J. Global and societal implications of the
diabetes epidemic. Nature 2001;414:782-7.
5. Manuel DG et al. Effectiveness and efficiency of different guidelines on statin
treatment for preventing deaths from coronary heart disease: modelling study. BMJ
2005;332:1419-22.
6. Eddy DM, Schlessinger L. Validation of the archimedes diabetes model. Diabetes
Care 2003;26:3102-10.
7. Hanley AJG et al. Prediction of type 2 diabetes using simple measures of insulin
resistance - Combined results from the San Antonio Heart Study, the Mexico City
Diabetes Study, and the Insulin Resistance Atherosclerosis Study. Diabetes 2003;52:463-
9.

8. Ito C et al. Prediction of diabetes mellitus (NIDDM). Diabetes Research and

Clinical Practice 1996;34:S7-S11.
9. Herman WH et al. A new and simple questionnaire to identify people at increased
risk for undiagnosed diabetes. Diabetes Care 1995;18:382-7.
10. Lindstrom J, Tuomilehto J. The diabetes risk score: a practical tool to predict type
2 diabetes risk. Diabetes Care 2007;26:725-31.
11. Larsson H et al. Prediction of diabetes using ADA or WHO criteria in post-
menopousal women: a 10-year follow-up study. Diabetologia 2000;43:279-88.
12. Stern MP, Williams K, Haffner SM. Identification of persons at high risk of type
2 diabetes mellitus: Do we need the oral glucose tolerance test. Annals of Internal
Medicine 2009;136:575-81.
13. Wilson P et al. Prediction of incident diabetes mellitus in middle-aged adults.
Archives of Internal Medicine 2007;167:1068-74.
14. McGeechan K et al. Assesing new biomarkers and predictive models for use in
clinical practice. Archives of Internal Medicine 2008;168:2304-10.

46
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

15. Ford ES, Giles WH, Dietz WH. Prevalence of the metabolic syndrome among US
adults - Findings from the Third National Health and Nutrition Examination Survey.
Jama-Journal of the American Medical Association 2002;287:356-9.
16. Hu FB et al. Diet, Lifestyle, and the Risk of Type 2 Diabetes Mellitus in Women.
New England Journal of Medicine 2001;345:790-7.
17. Statistics Canada. 1996-97 NPHS Public Use Microdata Documentation. 1999.
Ottawa.
18. Statistics Canada. Canadian Community Health Survey Methodological
Overview. Health Reports 2002;13:9-14.
19. Canadian Community Health Survey 2000–2001. Ottawa: 2003.
20. Hux JE, Ivis F. Diabetes in Ontario. Diabetes Care 2005;25:512-6.
21. Lipscombe LL, Hux JE. Trends in diabetes prevalence, incidence, and mortality
in Ontario, Canada 1995-2005: a population-based study. Lancet 2007;369:750-6.
22. Health Canada. Responding to the Challenge of Diabetes in Canada. 2003.
Ottawa, ON.
23. Statistics Canada. 1996-7 National Population Health Survey: Derived Variable
Specifications. 1999. Ottawa.
24. Odell PM, Anderson KM, Kannel WB. New Models for Predicting
Cardiovascular Events. Journal of Clinical Epidemiology 1994;47:583-92.
25. Farewell VT, Prentice RL. A study of distributional shape in life testing.
Technometrics 1977;19:69-75.
26. Pencina M, D'Agostino RB. Overall C as a measure of discrimination in survival
analysis: model specific population value and confidence interval estimation. Statistics in
Medicine 2004;23:2109-23.
27. Campbell G. General Methodology I: Advances in statistic methodology for the
evaluation of diagnostic and laboratory tests. Statistics in Medicine 2004;13:499-508.
28. D'Agostino RB et al. Validation of the Framingham Coronary Disease Prediction
Scores. JAMA 2001;286:180-7.
29. Nam B-H. Discrimination and Calibration in Survival Analysis: Extension of the
ROC Curve for Descrimination and Chi-square test for Calibration. 2000. Boston
University.
30. Brindle P et al. Predictive accuracy of the Framingham coronary risk score in
British men: prospective cohort study. BMJ 2003;327:1267.

47
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

31. Yeo D, Mantel H, Lui TP. Bootstrap variance estimation for the National
Population Health Survey. 778-783. 1999. Baltimore, American Statistical Association.
32. Kovacevic MS, Mach L, Roberts G. Bootstrap variance estimation for predicted
individual and population-average risks. Proceedings of the Survey Research Methods
Section. 2008. American Statistical Association.
33. Boyle JP et al. Projection of diabetes burden through 2050 - Impact of changing
demography and disease prevalence in the US. Diabetes Care 2001;24:1936-40.
34. King H, Aubert RE, Herman WH. Global burden of diabetes, 1995-2025 -
Prevalence, numerical estimates, and projections. Diabetes Care 1998;21:1414-31.
35. World Health Organization. Report of a WHO consultation on obesity, Obesity:
preventing and managing the global epidemic. 1998. Geneva, World Health
Organization.
36. Narayan KMV et al. Lifetime risk for diabetes mellitus in the United States.
JAMA 2003;290:1884-90.
37. Manuel D, Schultz S. Health-related quality of life and health-adjusted life
expectancy of people with diabetes in Ontario, Canada, 1996-1997. Diabetes Care
2004;27:407-14.
38. Forouhi NG et al. Diabetes prevalence in England, 2001 - estimates from an
epidemiological model. Diabetic Medicine 2006;23:189-97.
39. Mainous AG et al. Impact of the population at risk of diabetes on projections of
diabetes burden in the United States: an epidemic on the way. Diabetologia 2007;50:934-
40.
40. Ford ES et al. Explaining the decrease in US deaths from coronary disease, 1980-
2000. New England Journal of Medicine 2007;356:2388-98.
41. Altman DG, Royston P. What do we mean by validating a prognostic model?
Statistics in Medicine 2000;19:453-73.
42. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: Issues in
developing models, evaluating assumptions and adequacy, and measuring and reducing
errors. Statistics in Medicine 1996;15:361-87.
43. Carmelli D, Cardon LR, Fabsitz R. Clustering of Hypertension, Diabetes, and
Obesity in Adult Male Twins - Same Genes Or Same Environments. American Journal of
Human Genetics 1994;55:566-73.
44. DeFronzo RA, Ferrannini E. Insulin resistance. A multifaceted syndrome
responsible for NIDDM, obesity, hypertension, dyslipidemia, and atherosclerotic
cardiovascular disease. Diabetes Care 1991;41:173-94.

48
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

45. Lorenzo C et al. The metabolic syndrome as predictor of type 2 diabetes - The
San Antonio Heart Study. Diabetes Care 2003;26:3153-9.
46. Meigs JB et al. Risk variable clustering in the insulin resistance syndrome - The
Framingham Offspring Study. Diabetes 1997;46:1594-600.
47. Schmidt MI et al. Clustering of dyslipidemia, hyperuricemia, diabetes, and
hypertension and its association with fasting insulin and central and overall obesity in a
general population. Metabolism-Clinical and Experimental 1996;45:699-706.
48. Mainous AG et al. A Coronary heart disease risk score based on pateint-reported
information. The American Journal of Cardiology 2007;1236-41.
49. Ambler G, Brady AR, Royston P. Simplifying a prognostic model: a simulation
model based on clinical data. Statistics in Medicine 2002;21:3803-22.
50. Chambless LE et al. Prediction of ischemic Stroke Risk in the Atherosclerosis
Risk in Communities Study. American Journal of Epidemiology 2004;160:259-69.
51. Wilson PWF et al. Prediction of coronary heart disease using risk factor
categories. Circulation 1998;97:1837-47.
52. DeFronzo RA. Insulin Resistance, Hyperinsulinemia, and Coronary-Artery
Disease - A Complex Metabolic Web. Journal of Cardiovascular Pharmacology
1992;20:S1-S16.
53. Pepe MS et al. Limitations of the odds ratio in gauging the performance of a
diagnostic, prognostic, or screening marker. American Journal of Epidemiology
2004;159:882-90.
54. Flint E, Rimm E. Obesity and cardiovascular disease risk among the young and
old - is BMI the wrong benchmark? International Journal of Epidemiology 2006;35:187-
9.
55. Vazquez G et al. Comparison of body mass index, waist circumference, and
Epidemiology 2007;29:115-28.
56. Mason C, Katzmarzyk PT, Blair SN. Eligibility for obesity treatment and risk of
mortality in men. Obesity Research 2005;13:1803-9.
57. Bassuk SS, Manson JE. Epidemiological evidence for the role of physical activity
in reducing risk of type 2 diabetes and cardiovascular disease. Journal of Applied
Physiology 2005;99:1193-204.
58. Nawaz H et al. Self-reported weight and height: implications for obesity research.
Journal of Preventive Medicine 2001;20:294-8.
59. Rowland M. Self-reported height and weight. American Journal of Clinical
Nutrition 2007;52:1125-33.

49
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

60. Shields M, Gorber SC, Tremblay MS. Estimates of obesity based on self-report
versus direct measures. Health Reports 2008;19:1-16.
61. Bostrom G, Diderichsen F. Socioeconomic differentials in misclassification of
height, weight and body mass index based on questionnaire data. International Journal of
Epidemiology 1997;26:860-6.
62. Niedhammer I et al. Validity of self-reported weight and height in the French
GAZEL cohort. International Journal of Obesity 2000;24:1111-8.
63. Wardle K, Johnson F. Sex differences in the association of socioeconomic status
with obesity. Internation Journal of Obesity and Related Metabolic Disorders
2002;26:1144-9.

50
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3. Measurement in Risk Prediction Models

3.1 The influence of measurement error on accuracy (calibration), discrimination, and

overall estimation of a risk prediction model

3.1.1 Abstract

Background: Self-reported height and weight can be subject to several types of measurement

error. The impact of this measurement error on the estimated risk, discrimination, and calibration

of a model that uses height and weight expressed as body mass index (BMI) to predict 10-year

risk of developing diabetes incidence has never been systematically studied.

Objective: To use simulation to quantify and describe the effect of random and systematic

measurement error in self-reported height and weight on the performance of a model for

predicting diabetes. The three performances measures used are predicted risk, accuracy, and

discrimination.

Methods: Predicted risk, Hosmer-Lemeshow goodness-of-fit χ2, correlation between observed

and predicted probabilities of developing diabetes and C statistic were measured under various

levels of random and systematic error. Simulations were done 500 times with sample sizes

typical of population-bases surveys (~10,000) used to collect self-reported risk factor

information. Results were run separately for males and females.

Results: Simulation data successfully reproduced estimates, discrimination and calibration

values similar to those generated by the algorithm that was derived from actual population data.

Increasing levels of random error in height and weight reduced the calibration and discrimination

of the risk algorithm. Furthermore, random error biased the predicted risk upwards. Systematic

51
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

error biased predicted risk in the direction of the under- or over-estimation and reduced

calibration; however, it did not affect discrimination.

Conclusions: This study demonstrates that random and systematic errors have the potential to

influence the performance of risk algorithms. Overall predicted risk should be carefully

considered in the context of potential measurement errors. Further research that quantifies the

amount and direction of error can improve the performance of prediction tools by allowing for

adjustments in exposure measurements.

52
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.1.2 Introduction

Epidemiologic studies rely on the measurement of exposure and outcome variables.

Measurement error is said to occur when the measured value of a variable does not equal the

‗true‘ value. Measurement error is a major concern in epidemiologic research and has been

discussed extensively in the literature(1-5). In general there are two classes of measurement

error: (i) random error is error that on average is equal to zero (ii) systematic error is error that on

average is not equal to zero (6). Measurement error has been mainly examined with respect to its

effect on risk estimates, such as risk ratios or hazard ratios(1, 7, 8). This research has led to

improvements in the critical appraisal and interpretation of epidemiologic findings and has

provided guidance for measurement in future studies. While useful for understanding the effects

of error on etiological estimates of disease, the findings from these studies do not directly apply

to risk algorithms.

Risk prediction is a key aspect of clinical work and has been recently has been applied to

population health through the Diabetes Population Risk Tool (DPoRT) (Chapter 2). Accurate and

valid prediction of the probability of developing a disease given a set of baseline risk factors is a

valuable tool for clinical management, health policy and population health decision making. In

risk algorithms disease prediction is based on a set of baseline variables that may contain

measurement error that could affect the prediction, discrimination and accuracy of the risk tool.

DPoRT is made more accessible by using characteristics routinely measured in population

health surveys to make predictions. National health surveys are conducted through telephone

interviews and are based on self-reported responses. These self-reported responses can result in

random error due to imperfect recall or misunderstanding of the question. They can also result in

systematic error as a result of psychosocial factors, such as social desirability. The influence of

53
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

error from self-report surveys on disease prediction has not been systemically studied. In

particular, the influence that measurement error has on predictive accuracy has never been

examined. By understanding the consequence of measurement error on risk algorithms, efforts

could be made to correct for these errors and thus improve the accuracy and validity of a risk

algorithm. Furthermore, other developers of population risk tools can use this information to

better weigh the pros and cons of using different types of data.

The objective of this study is to use simulation to understand the effect of measurement

error in self-reported risk factors on the performance of a simple risk algorithm to predict

diabetes. This study will focus on the measurement of Body mass index (BMI), a function of

height and weight, because it has the greatest influence on diabetes risk (9-13). In general, there

is a high correlation between self-report and measured height and weight (14); however, a recent

systematic review to summarize the empirical evidence regarding the concordance of objective

and subjective measures of assessing height and weight found that the general trend was an

underestimation of weight and an overestimation of height (15). Currently, there are no linked

data available that allow assessment of self-reported versus measured or ‗true‘ BMI

simultaneously with the diabetes outcome. Simulation allows one to examine the impact of

varying levels of measurement error on discrimination and accuracy. These results will inform

DPoRT and assist in understanding how its predictions and validation are affected by the use of

population health surveys.

Research from studies that examine the impact of measurement error on relative risk

shows that, in general, random error results in an estimate that is closer to the null value; whereas

systematic error results in over- or underestimates of risk in a particular direction depending on

54
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

the nature of the bias (1, 4, 7). These findings were the basis for the following a priori

hypotheses:

(i) Random (random) measurement error will affect both discrimination and calibration of a

model. The presence of measurement error will increase the observed variance in BMI and thus

widen the distribution of BMI in both diabetics and non-diabetics. This will affect discrimination

by increasing the overlap between the BMI distribution of those that are likely to develop

diabetes and the distribution of those that are not. This increased overlap will make it more

difficult to resolve them into populations of diabetics and non-diabetics. Calibration will be

affected because the predicted risk estimate will be higher or lower than the true estimate, due to

the over- or under-estimation of weight and height, and will result in misclassification of

diabetes status, resulting in a lower concordance with the observed estimate.

(ii) Systematic error will have minimal affects on discrimination but significant effects on

calibration of a model. Systematic error can result in an average over- or under-estimation of

predicted risk; however, this should not influence the ability to rank order subjects, under the

assumption that the error is not differential between persons that develop or not develop the

disease. The same rationale outlined in (i) applies to calibration under this scenario in that the

over- or under-estimation of weight and height will reduce the concordance with the observed

estimate. However the effect will be more apparent here since the error is occurring all in one

direction.

55
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

(iii) Random error will not affect the overall predicted risk of the model and systematic error

will influence the predicted risk in the direction of the systematic error. Random error increases

the variability, which increases the range of predicted risk equally in both directions and thus,

will not skew the distribution in any particular direction. Therefore, the overall predicted risk

estimate should be similar to the predicted risk estimate from the model without measurement

error.

56
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.1.3 Methods

Simulating Data

Data to simulate the relationship of a continuous variable (BMI) related to probability of

developing diabetes were created to closely match the actual data used to create DPoRT. For

this simulation, the algorithm predicting the binary outcome of physician-diagnosed diabetes was

modeled using a logistic regression form equation and included BMI and its quadratic term as

predictor variables. The model can be described as follows:

Logit(p) = β0 + β1*BMI + β2*BMI2

where p represents the probability of developing BMI in the next 10 years and is calculated as:

P(Yi = 1|Xi) = exp(β‘Xi) / ( 1 + exp(β‘Xi) ) and P(Yi = 0|Xi) = 1 / ( 1 + exp-(β‘Xi) )

where =i, ………n

β‘ represents the vector of parameter estimates estimated by maximum likelihood

X represents the vector of fixed covariates for each subject

To randomly generate a person with diabetes, first height (m) and weight (kg) are generated as

Gaussian random variables with means and variances obtained from DPoRT development

cohort. Using the specified values for the regression coefficients β0, β1 and β2 the logit was

calculated from which the probability P of the person having diabetes is calculated. Then a 0 / 1

Bernoulli random variable with probability P is generated with 1 or 0 meaning that the person

does or does not have diabetes.

BMI is calculated as the ratio of height in meters (m) to weight in kilograms (kg)

squared. Both height and weight were varied individually from the true value assuming different

levels of error in the observed estimate. The size of the random measurement error is defined by

the intraclass correlation coefficient (ICC), the proportion of the overall observed variance

57
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

attributable to the ―true‖ variance between subjects. The systematic levels of measurement error

in height and weight were taken from a recent systematic review that summarized the empirical

evidence regarding the concordance of objective and subjective measures of height and weight

(16). These error values are consistent with a recent study examining the difference between

measured and self-reported height and weight in the CCHS (17). The observed mean and

standard deviation of BMI, taken from the National Population Health Survey (NPHS) (18),

along with the parameter estimates from the linkage with the Ontario Diabetes Database (ODD)

(19, 20) (See chapter 2) were used in our simulation. This simpler algorithm that included only

BMI and not the entire set of algorithm variables used in DPoRT was chosen because it has been

shown that it explains the majority of variation in diabetes incidence across individuals in a

population and produces C statistics as high as 0.73. For this study, the simulation data were

created such that the discrimination value was equal to that observed in the actual DPoRT

development cohort ~ 0.7. The simulated data were created using observed data assuming no

measurement error and then re-fit using various values of systematic and random error in height

and weight. .

58
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Random Error

The observed variance of height and weight can be described as follows:

(I) σ2observed = σ2true + σ2error

where:

σ2true is the true variance of the measurement

σ2error is the variance, which is attributed to measurement error and σ2observed ≥ σ2true

By these definitions, in the case where no error exists σ2observed = σ2true the intraclass correlation

coefficient (ICC), denoted as ρ, is an estimate of the fraction of the total measurement variance

associated with the true variation among individuals (21, 22).

σ 2true
(II) ρ =
σ 2true + σ 2error

From (I) we can express σ2true as σ2true = σ2observed - σ2error and substituting this expression back

into (II) gives us:

ρ = (σ2observed - σ2error) / ((σ2observed - σ2error)+ σ2error

and therefore ρ can be expressed as:

σ 2observed − σ 2error
(III) ρ =
σ 2obs erved

Re-arranging (III) allows us to express σ2error as a function of ρ:

59
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

(IV) σ2error = σ2observed(1- ρ)

If no error exists in the measurement ρ = 1.0 and σ2error = 0 and from (I) the σ2observed = σ2true. In

other words, all the observed variance is due to the true variance among individuals. An example

of the influence of ICC on the distribution of BMI can be seen in the appendix section 6.4.

Systematic Error

Deviation of average self-reported height and weight can be represented as:

(V) 𝑌𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 = 𝑌𝑡𝑟𝑢𝑒 + ωi

where:

𝑌𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 = the average self-report response in the survey

𝑌𝑡𝑟𝑢𝑒 = the average true value of the response in the survey

ωi represents the average amount of over- or under- estimation from the true value

Traditionally the E(ωi) = 0 and thus the given measurement is, on average, equal to the reported

measurement; however, in this simulation ωi will represent various levels of over- and

underestimation.

60
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Correlation of Height and Weight

In the simulation, the errors associated with self-reported height and weight will be varied

independently; however there is a strong correlation between height and weight that must be

considered. All starting correlation values were taken from the original data. The following

definition will be used to express the correlation between height and weight.

Let be:

rhw(observed) = observed correlation coefficient between height and weight

A1 = σheight

B1 = σweight * rhw(observed)

B2 = (σ2weight − B12 )

B1 and B2 will be used to define the variance in weight, such that B1 defines the portion of the

variance in weight that is related to its correlation of height.

The correlation of height and weight will vary according to ICC values such that

rhw for the true height and weight under measurement error was defined as:
rhw(observed)
rhwTRUE =
𝐼𝐶𝐶ℎ𝑒𝑖𝑔ℎ𝑡 × 𝐼𝐶𝐶𝑤𝑒𝑖𝑔ℎ𝑡

61
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Therefore for the observed height and weight:

Let be:

M1 = A1 x Z1

M2 = B1 x Z1 + B2 x Z2 or

M2 = B1 x Z1 + (σ2weight observed − B12 ) x Z2

The correlation is a function of the B2, which is common to both height and weight

Z1 and Z2 are independent standard normal random variables

The true height and weight values were generated using the same approach except rhw(true) will

be used instead of rhw(observed).

62
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Simulation using various levels of ICC and systematic error

Simulated values of height and weight were generated by applying a random component to the

standard deviation of the variable. By applying the definitions of M1 and M2 above and using

the following definitions:

𝐻𝑒𝑖𝑔 ℎ𝑡 𝑖 𝑊𝑒𝑖𝑔 ℎ𝑡 𝑖
𝐻𝑒𝑖𝑔ℎ𝑡 = and 𝑊𝑒𝑖𝑔ℎ𝑡 =
𝑛𝑖 𝑛𝑖

where i = 1 ….. n for each individual in the cohort

Height = 𝐻𝑒𝑖𝑔ℎ𝑡 + M1

Weight = 𝑊𝑒𝑖𝑔ℎ𝑡 + M2

𝑊𝑒𝑖𝑔 ℎ𝑡
BMI =
𝐻𝑒𝑖𝑔 ℎ𝑡 2

BMIOBSERVED2 = BMIOBSERVED X BMIOBSERVED

*note that when using ICC = 1.0 it follows that σ2observed = σ2true, which is the way the data are

currently used when applying DPoRT. This analysis quantified the influence of random error by

simulating scenarios where a proportion of the observed variance was attributed to random error.

These steps were replicated for the given sample (9,177 for males and 10,618 for

females) 500 times. Ten-year cumulative incidence was maintained between 7—10 %, in

accordance with Ontario male and female diabetes rates taken from the Ontario Diabetes

Database. The model was first fit assuming that there was no error in the observed height and

weight values to ensure that the expected percent rejection of the null hypothesis, which would

be equal to the type I error rate (or α) under the null distribution is equal to 5% and that the

63
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

average C statistics were consistent with those seen in the DPoRT development cohort. All

simulations were done using SAS statistical software (version 9.1 SAS Institute Inc, Cary, NC)

and random numbers were generated using the RAN family of functions (RANUNI and

RANNORM).

Calibration and Discrimination

Calibration was measured using the Hosmer-Lemeshow goodness of fit statistic (χ2H-L) where

observed and expected values are compared across deciles of risk (23-25). This statistic is

computed by dividing the validation cohort into deciles of predicted risk. Consistent with

D‘Agostino‘s approach for evaluating observed and predicted values using risk algorithms the

value 20, the 99th percentile of a chi square with 8 degrees of freedom , was used as a cutoff to

mark sufficient calibration (26). Power to detect calibration was therefore defined, as the

proportion of simulations out of the 500 simulations that achieved the χ2H-L = 20 cutoff. This

proportion was calculated under the various error situations. Discrimination was measured using

a C-statistic, analogous to the area under the ROC curve (27). The C-statistic was calculated

using rank statistics and verified by the output from the LOGISTIC procedure. A study flow

diagram and an example of the SAS code for females can be seen in the Appendix section 6.4.

64
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Assumptions

For this study the following assumptions were made:

(i) The random or systematic error does not differ between persons that develop and do not

develop the disease or other subpopulations.

(ii) Other sources of error including error in ascertainment of diabetes status, selection bias in the

survey, and/or sampling are assumed to be absent.

65
 Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.1.4 Results

The fitted risk algorithms for males and females are shown in Table 1. These algorithms predict

the probability of developing diabetes in 10 years given the respondents‘ BMI, which is a

function of their reported height and weight.

Table 1a. Starting values taken from 1996-7 National Population Health Survey used in
simulation.
Parameter Males Females
mean (standard deviation) (N = 9, 177) (N = 10, 618)
Height (m) 1.768 (0.075) 1.627(0.069)
Weight (kg) 81.624 (13.805) 64.761 (12.320)
BMI (m/kg2) 26.076 (3.995) 24.495 (4.586)
Correlation for rhw = 0.475 rhw = 0.311
height and weight (rhw)
10-year DM incidence 9.17% 7.35%

Table 1b. Properties of actual risk equation relating BMI to probabilities of developing diabetes
from NPHS cohort and values from the simulation model.

Males – NPHS data (N = 9, 177) Males – Simulation (N = 9, 177)

Variable Coefficient Standard Error Pvalue Coefficient Standard Error Pvalue
BMI 0.4202 0.0383 <0.0001 0.4263 0.0111 <0.0001
BMI2 -0.00437 0.000618 <0.0001 -0.00448 0.001049 <0.0001
Intercept -10.4034 -10.4897
Model Properties
Calibration (χ2HL) χ2HL = 5.67, p-value = 0.6841 χ2HL =9.951, p-value = 0.3689
Discrimination (C-statistic) C = 0.677 C = 0.686
Females – NPHS data (N = 10, 618) Females – Simulation (N = 10, 618)
Variable Coefficient Standard Error Pvalue Coefficient Standard Error Pvalue
BMI 0.4565 0.0554 <0.0001 0.4593 0.0779 <0.0001
BMI2 -0.00509 0.00091 <0.0001 -0.00514 0.00141 <0.0001
Intercept -10.8967 -10.5899
Model Properties
Calibration (χ2HL) χ2HL = 9.33, p-value = 0.3153 χ2HL = 10.466, p-value = 0.3356
Discrimination (C-statistic) C = 0.726 C = 0.718

66
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

The results from the simulations are presented by error type (random and systematic error) and

by the influence on three areas: predicted risk; calibration and validation. Due to the amount of

data, most results are displayed in graphical form; however tables of the results can be found in

appendix 6.9.

Random Error

Random error in height and weight were examined for ICC = 1.0, 0.9, 0.8, 0.7, 0.6 and 0.5.

Impact on Predicted Risk

Under the presence of random error the overall probability of developing diabetes predicted from

the risk algorithm was higher than the estimate from the algorithm applied to the data without

random error. In other words, the presence of random error biased the overall predicted risk

estimate upwards. The differences between the predicted risk without error and with error were

relatively small, with the biggest differences being 0.99% higher than the truth (N = 90 more

cases) for males and 0.89% higher for females (N = 95 more cases) under extreme levels of

random error. Random error in weight had a bigger influence on the predicted risk than random

error in height. When the ICC for weight was held at 1.0 (i.e. no error in weight) but ICCs in

height were allowed to vary (from 1.0 to 0.5), the largest overestimate in diabetes risk was 0.30%

(28 more cases) in males and 0.16% (70 more cases) in females. However, when the ICC for

height was held at 1.0 but the ICCs for weight were allowed to vary, the largest overestimate was

0.70% (64 more cases) for females and 0.66% (39 more cases) for males (Figure 1). In the

presence of random error, the observed distribution of predicted risk is skewed right compared to

the true distribution. Figure 2 shows the observed distribution of predicted risk under the

assumption of no random error (thus reported values = true values) compared with the true

67
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

distribution of predicted risk if the observed data had random error (ICC values of 0.8 and 0.6).

The fact that the true distributions have a narrower range relative to the observed distributions

indicates that if the data to which the algorithm was applied contained some level of random

error then the true distribution of predicted probability of developing diabetes would be narrower

than what is estimated from the algorithm. Further, the effect on the distribution largely

influences the right side of the distribution since the left side is bounded by zero.

Impact on Calibration

Under the assumption of no random error in the observed data, the χ2H-L value was < 20

(calibration cutoff achieved) 97% of the time for males and females. At increasing levels of

random error the proportion of simulations where H-L χ2H-L was less than the cutoff 20 decreased

steadily. Overall, ICCs of approximately 0.8 or higher resulted in the algorithm achieving the

calibration criteria, that is a H-L chi square less than 20, at least 80% of the time. In both males

and females, errors in weight lead to larger decreases in calibration than errors in height, such

that even a perfect height measurement (ICC height = 1.0) would result no calibration if the ICC

for weight drops below 0.8. On the other hand, if ICC for weight was 1.0 even if the ICC for

height was 0.6, the algorithm could still achieve calibration almost 80% of the time (Figure 3).

Impact on Discrimination

Discrimination was decreased in the presence of random error. Under the most severe

measurement error the C-statistic was reduced from a C-statistic of 0.69 (with no error) to 0.55 in

males and from 0.72 (with no error) to 0.63 in females. If the ICCs for height and weight were

higher than 0.8, then the differences in the C-statistic compared to the estimate that had no

68
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

random error were less than 0.02. As with calibration, error in weight had a bigger impact on the

C-statistic than errors in height (Figure 4).

69
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 1. Difference in overall risk (observed – true) under random error for males and females
(averaged over 500 simulations).

70
Figure 2.Observed (top graphs) and true (middle and bottom graphs) distribution of the probability of developing diabetes under
increasing levels of random error (ICC = 0.8 and 0.6) compared with observed distribution.
Observed Distribution
ICC = 1.0

Mean: 7.20%
22.5

20.0

17.5

15.0

Min, Max: (0.04%, 33.49%)

12.5
Percent

10.0

7.5

5.0

90th percentile: 7.38%

2.5

0
0.005 0.025 0.045 0.065 0.085 0.105 0.125 0.145 0.165 0.185 0.205 0.225 0.245 0.265 0.285 0.305 0.325 0.345

prob

True distribution when ICC

ICC ==0.8
0.8
22.5

20.0
Mean: 7.00%
17.5

Min, Max: (0.01%, 32.65%)

15.0

12.5
Percent

10.0

7.5

90th percentile: 7.06 %

5.0

2.5

0
0.005 0.025 0.045 0.065 0.085 0.105 0.125 0.145 0.165 0.185 0.205 0.225 0.245 0.265 0.285 0.305 0.325 0.345

Mean: 6.80%
prob
True distribution when ICC
ICC ==0.6
0.6
25

20

15 Min, Max: (0.01%, 31.98%)

Percent

10

90th percentile: 6.85 %

0
0.005 0.025 0.045 0.065 0.085 0.105 0.125 0.145 0.165 0.185 0.205 0.225 0.245 0.265 0.285 0.305 0.325 0.345

prob
Figure 3a. Ability to detect calibration in 500 replications under various levels of random error in
height and weight among males (top) and females (bottom).
Figure 4. Average C-statistic for 500 replications under various levels of random error in height and weight.
Systematic error

The values for systematic measurement error in height and weight used in this study were taken

from a recent systematic review that summarized the empirical evidence regarding the

concordance of objective and subjective measures of height and weight (16). The under-

reporting of weight was varied from 0 kg to 3.0 kg below true value (varied in increments of 0.5

kg). The over-reporting of height was varied from 0 cm to 3.0 cm above the true value in

increments of 0.5 cm. The average level of systematic error found in the systematic review was

an under-reporting of weight of 1.7 kg and an over-reporting of height of 2.5 cm.

Impact on Predicted Risk

As expected, under-reporting of weight and over-reporting of height resulted in an underestimate

of predicted probability of developing diabetes mellitus. Expected levels of under-reporting of

height and weight taken from the systematic review (+2.5 cm and -1.7 kg) would result in 0.86%

reduction (91 fewer cases) in the overall predicted probability of developing diabetes for females

and a 0.91% reduction (84 fewer cases) for males (Figure 5). The presence of random error in

conjunction with systematic error slightly reduced the amount of underestimation (Figure 6).

Impact on Calibration

Overall, under-reporting of weight of at least 1.7 kg or over-reporting of height of at least 1.5cm

resulted in the algorithm achieving the calibration at least 80% of the time. None of the 500

simulations achieved calibration under the maximum biases in reported height and weight

(Figure 7). It must be noted that there is no evidence that these extreme biases are likely in self-
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

reported height and weight; rather, they were investigated to illustrate of the range of results of

under- and over-reporting. The presence of random error in conjunction with systematic error

did not significantly worsen or improve power to detect calibration.

Impact on Discrimination

Under- or over-reporting of weight and height did not have a significant effect on the

discrimination of the model (Figure 8). C-statistics were reduced very slightly in the most

extreme case of under-reporting of weight or over-reporting of height; however, the difference

from the true estimate was never more than 0.01 for both males and females. When both random

error and systematic error were imposed, the C-statistic was reduced; however this was due to

the influence of random error and not systematic error.

75
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 5. Difference in overall risk (observed – true) under systematic error for males and
females (averaged over 500 simulations).

76
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 6. Difference in overall risk (observed – true) under systematic error and random error in
height and weight for males and females (averaged over 500 simulations).

77
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 7. Ability to detect calibration in 500 replications under various levels of systematic error
in males and females.

Percent that achieved calibration in H-L test among males in the presence of bias - 500 Replications

Males Females
100%

90%

80%

70%

60%
% H-L <20

50%

40%

30%

20%

10%

0%
None "-0.5 kg" "-1.0 kg" "-1.5 kg" "-2.0 kg" "-2.5 kg" "-3.0 kg" "+0.5 "+1.0 "+1.5 "+2.0 "+2.5 "+3.0
cm" cm" cm" cm" cm" cm"

Bias

78
Figure 8. Average C-statistic for 500 replications under various levels of systematic error.
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Results in conjunction with a priori hypotheses

Below is a summary of the a priori hypotheses and findings of the study in response to those

hypotheses:

(i) Random measurement error will affect both discrimination and calibration of the risk

algorithm.

This was confirmed in this study. Increasing levels of random error (generally ICCs below 0.8)

reduced the discrimination and calibration compared to a model without random error.

(ii) Systematic error will have minimal affects on discrimination but large effects on calibration

of the risk algorithm.

This was confirmed in this study. Systematic error resulted in reductions in calibration;

however, even in extreme situations of under-reporting of weight or over-reporting of height the

discrimination (C-statistic) remained unaffected.

(iii) Random error will not affect the overall predicted risk of the model and systematic error

will influence the predicted risk in the direction of the under- or over-estimation of height and

weight.

This hypothesis was only partly confirmed in this study. It was confirmed that systematic error

influenced the predicted risk in the direction of the systematic error. However, contrary to this

hypothesis, random error also biased the overall estimate of predicted risk and the direction of

this bias for this scenario was upward.

80
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.1.5 Discussion

This study systematically examined the impact of measurement error in the context of a

prediction algorithm. Simulation studies are flexible and permit a range of assumptions about the

magnitude and direction of errors. This simulation study reveals several interesting aspects of the

influence of measurement error (systematic and random) on the performance of a risk algorithm.

Two out of the three a priori hypotheses were realized in this study.

As hypothesized, random error reduced calibration and discrimination of the algorithm.

From equation (I) σ2observed = σ2true + σ2error , it can be seen that that the observed variance is

greater than the true variance in the presence of measurement error. In the context of Canada‘s

population health surveys, the observed variance of self-reported BMI would be greater than the

true variance because self-reported BMI may contain some random error. This study confirmed

that measurement error increases the observed variance of BMI, thus, making the observed BMI

distribution from the public use data wider than the true distribution. This affects both diabetics

and non-diabetics resulting in greater overlap between the BMI distributions making assigning

risk according to different levels of BMI more difficult to achieve. Even though random error in

height and weight should on average correctly estimate the true BMI in the population (since it

does not skew the mean in a particular direction), it can still influence the performance of a

prediction model due to decreased precision, which leads to greater dispersion in the BMI

distribution.

This study confirmed that systematic error in height and weight will result in bias in the

predicted risk estimates. This affects calibration, which is not surprising since the concordance

of observed and predicted events would be influenced by the under- or over-reporting of the

level BMI. In other words, persons that are over- or under-reporting their weight will then be

81
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

over- or under-estimated by the risk model and thus result in disagreement with observed

estimates. As hypothesized, systematic error did not influence the ability to rank order subjects.

In other words, the ability to discriminate between who will and will not develop diabetes was

not affected by systematic error, when variance due to random error is held constant. This was

reflected by the stability of the C-statistic under varying degrees of systematic error. The way

that the systematic error was examined in this study was such that the distribution of BMI was

shifted to the left (as a result of underestimating weight or overestimating height or both)

compared to the true distribution. This is an overall effect and the decreased precision or

increased variability as seen with random error is not observed. Even though the distribution is

shifted to the left – those with higher BMI still have a higher probability of developing diabetes

compared to those with lower BMI, even though the absolute levels of risk will be under-

estimated in both groups. This is a classic example of how discrimination and calibration are

often discordant. Due to the nature of probability, it is possible for a prediction algorithm to

exhibit perfect discrimination – i.e. it can perfectly resolve a population into those who will and

will not experience the event - and at the same time have deficient accuracy (meaning that the

predicted probability of that person experiencing the event does not agree with what will be

actually observed) (28). This study did not impose systematic error with respect to disease

status, but it could be hypothesized that if the systematic error were differential between

diabetics and non-diabetics that this could indeed affect discrimination. This is a topic of future

research.

The finding that random error resulted in the overall predicted risk estimated to be biased

upwards was contradictory to the hypothesis that only systematic error will bias the risk estimate.

As mentioned, random error increases the variability of a measurement and increases the range

82
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

of predicted risk. From other simulations not reported here it was shown that that if the true

prediction was greater than 0.5 then the estimated probability based on variables measured with

error would be smaller than the observed probability. That is, in both cases the prediction using

variables with measurement error pushes the estimated probabilities closer to 0.5

Not surprisingly, the error in predicted risk resulting from under-reporting weight or

over-reporting height is in the anticipated direction i.e. if weight is under-reported the observed

risk will be underestimated and not surprisingly, based on the above discussion, the addition of

random error to this type of systematic error slightly reduced the amount of underestimation

because the random and systematic errors work in opposite directions. In another situation,

random error could potentially augment the error in predicted risk. Such would be the case if

systematic error tended to result in an overestimate of risk.

There are several results from this study that have implications for DPoRT. DPoRT

relies on self-report survey data, which is likely to suffer from some form of random error. This

study confirms that random error, which makes up to 20% of the total observed variance (ICC of

0.8 or higher), is unlikely to affect the performance or validation of the model. Research shows

that the random error in height and weight reporting is unlikely to exceed that amount (14).

However, the level of random error in the self-reporting of height and weight in the national

health surveys need to be confirmed to ensure that it does not make up more than 20% of the

total observed variance. Interestingly, the effects on the estimate of predicted diabetes risk in the

population were relatively minor, even in situations of high under- or over- reporting of weight

and height. This is likely because BMI has such a strong relationship with the outcome of

diabetes such that increased risk is apparent even with significant underestimation. In other

words, the true distributions of BMI in diabetics and non-diabetics are so distinct that even in the

83
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

presence of underreporting these populations has dissimilar risk for developing diabetes. Had

this misclassification affected a variable which did not have such a strong relationship with the

outcome, the effect on predicted risk may have been more severe. Furthermore, in this study

systematic error in self-reported height and weight was taken as an overall effect in the

population. If self-reporting error were significantly more likely to occur in those who were

more likely to develop diabetes then the impact of this bias could be augmented. This is another

topic for future research.

This study examined a range of error values found from validation studies looking at self-

report and measured height and weight compared with measured height and weight. A recent

study by Shields et al. (17) examined agreement between self-report and measured BMI in a sub-

sample of the CCHS population. Overall systematic error in females was +0.5 cm for height and

-2.5 kg for weight which corresponded to an average underestimate of BMI of 1.2 kg/m2. In

males the bias was +1.0 cm for height and -1.8 kg for weight which corresponds to an average

underestimate of BMI of 0.9 kg/m2. According to these values, DPoRT predicted risks may be

underestimated by ~1% for both sexes; however, this underestimation is in the context of no

systematic error, which as discussed above may minimize underestimation. The relatively small

amount of underestimation in predicted risk that occurs due to systematic error in self-reported

height and weight is almost exactly proportional to the magnitude of error i.e. BMI is

underestimated by ~1 (0.9 & 1.2 for males and females) and overall predicted risk is

underestimated by ~1 % in males and females.

If DPoRT were applied in a setting where reporting error is thought to be higher than in

the NPHS (where the tool was developed), this increased error must be taken into account when

predicting risk and validating the tool. If the amount of error can be quantified, this study

84
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

outlines a way to correct predicted risk estimates using the ICC and equation (IV) σ2error =

- ρ). Since this study shows that systematic error has the ability to influence the

discrimination and accuracy of a risk algorithm performance, research into understanding and

quantifying these errors could potentially improve the performance of DPoRT.

This study focused on the overall trend of self-reporting error seen in several validation

studies, that is an underestimation of weight and an overestimation of height (15); however these

patterns may also vary across subpopulations such as gender and socioeconomic status.

Generally women tend to underestimate weight more so than men and men tend to overestimate

height more so than women(14, 29). Socioeconomic status has been shown to modify these

associations such that those of lower socioeconomic status may actually overestimate their

weight and/or underestimate their height (30, 31). These subgroups may also have differential

diabetes risk and the effect to which this error influences population risk prediction is a topic of

future research.

There are several limitations to consider in the context of this study. Conclusions drawn

from this simulation study will relate only to the scenarios simulated and may not apply to all

risk algorithm situations. Simulation programs that reflect the specific study conditions to which

a study is applied to must be created to make conclusions applicable. Another caution in

interpreting the findings of this study is that models examined in this exercise are simpler than

complicated multivariate risk algorithms encountered in practice. This simpler model allows us

to focus on the height and weight error, which is the greatest potential source of error in DPoRT.

It should be noted that one of the assumptions of this study is that the only sources of error are

that in self-reported height and weight. Other sources of error including, error in diabetes status

and selection bias in the survey or in sampling, are assumed to be absent. We cannot confirm

85
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

that the results of this study would be the same in the presence of the above mentioned sources of

error.

This study has provided novel information about the influence of measurement error in a

prediction model. By understanding the consequences of measurement error on prediction and

algorithm performance, efforts can be made to correct for these errors and thus improve the

accuracy and validity of a risk algorithm. Future research will include investigation into

systematic error with respect to disease status or other characteristics of the population. Further,

efforts must be made to understand the nature of error in self-reporting measurements. Ongoing

work to improve the quality of measurements used in risk algorithms will improve model

performance. Researchers developing and validating risk tools must be aware of the presence of

measurement error and its impact on the performance of their risk tools.

86
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.16 References

1. Flegal KM, Keyl PM, Nieto FJ. The effects of exposure misclassification on
estimates of relative risk. Epidemiology 1986;123:736-51.

2. Greenland S. The effect of misclassification in the presence of covariates.

American Journal of Epidemiology 1980;112:564-9.

3. Wacholder S. When measurment errors correlate with truth: Suprising effects of

nondifferential misclassification. Epidemiology 2007;6:157-61.

4. Willet WC. An overview of issues related to the correction of non-differential

errors in exposure measurement. Statistics in Medicine 1989;8:1040.

5. Wong JVA, Le ND, Burnett R. Causality, Measurement error and

multicollinearity in epidemiology. Environmetrics 1996;7:441-51.

6. Last JM. A Dictionary of Epidemiology. Oxford: Oxford University Press, 2001.

7. Fuller WA. Estimation in the presence of measurement error. International

Statistical Review 1995;63:121-41.

8. Weinstock MA, Colditz GA, Willet WC. Recall (report) bias and reliability in the
retrospective assessment of melanoma risk. American Journal of Epidemiology
1991;133:240-5.

9. Colditz G et al. Weight as a risk factor for clinical diabetes in women. American
Journal of Epidemiology 1990;132:501-13.

10. Colditz G et al. Weight gain as a risk factor for clinical diabetes mellitus in
women. Annals of Internal Medicine 1995;122:481-6.

11. Perry IJ et al. Prospective study of risk factors for development of non-insulin
dependent diabetes in middle aged British men. British Medical Journal 1995;310:555-9.

12. Vanderpump MPJ et al. The incidence of diabetes mellitus in an English

community: a 20-year follow-up of the Wickham Survey. Diabetic Medicine
1996;13:741-7.

13. Wilson P et al. Prediction of incident diabetes mellitus in middle-aged adults.

Archives of Internal Medicine 2007;167:1068-74.

14. Nawaz H et al. Self-reported weight and height: implications for obesity research.
Journal of Preventive Medicine 2001;20:294-8.

15. Gorber SC et al. A comparison of direct vs. self-report measures for assesing
height, weight, and body mass index: a systematic review. Obesity Reviews 2007;8:307-
26.
87
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

16. Gorber SC et al. The feasibility of establishing correction factors to adjust self-
reported estimates of obesity. Health Reports 2009;19.

17. Shields M, Gorber SC, Tremblay MS. Estimates of obesity based on self-report
versus direct measures. Health Reports 2008;19:1-16.

18. Statistics Canada. 1996-97 NPHS Public Use Microdata Documentation. 1999.
Ottawa.

19. Hux JE, Ivis F. Diabetes in Ontario. Diabetes Care 2005;25:512-6.

20. Lipscombe LL, Hux JE. Trends in diabetes prevalence, incidence, and mortality
in Ontario, Canada 1995-2005: a population-based study. Lancet 2007;369:750-6.

21. Deyo RA, Diehr P, Patrick DL. Reproducibility and responsiveness of health
status measures: Statistics and strategies for evaluation. Controlled Clinical Trials
2008;12:142S-58S.

22. Fleiss J. Statistical Methods for Rates and Proportions. New Jersey: John Wiley &
Sons, 2003.

23. Hosmer DW, Lemenshow S. Applied Logistic Regression. New York: Wiley,
1989.

24. Hosmer DW et al. A comparison of goodness-of-ft tests for the logistic regression
model. Statistics in Medicine 1997;16:965-80.

25. Hosmer DW, Lid Hjort N. Goodness-of-fit processes for logistic regression:
simulation results. Statistics in Medicine 2002;21:2723-38.

26. D'Agostino RB et al. Validation of the Framingham Coronary Disease Prediction

Scores. JAMA 2001;286:180-7.

27. Campbell G. General Methodology I: Advances in statistic methodology for the

evaluation of diagnostic and laboratory tests. Statistics in Medicine 2004;13:499-508.

28. Diamond GA. What price perfection? Calibration and discrimination of clinical
prediction models. Journal of Clinical Epidemiology 1992;45:85-9.

29. Niedhammer I et al. Validity of self-reported weight and height in the French
GAZEL cohort. International Journal of Obesity 2000;24:1111-8.

30. Bostrom G, Diderichsen F. Socioeconomic differentials in misclassification of

height, weight and body mass index based on questionnaire data. International Journal of
Epidemiology 1997;26:860-6.

88
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

31. Wardle K, Johnson F. Sex differences in the association of socioeconomic status

with obesity. Internation Journal of Obesity and Related Metabolic Disorders
2002;26:1144-9.

89
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.2 The role of ethnicity in the population-based prediction of diabetes

3.2.1 Abstract

Background: Certain ethnic groups have been shown to be at increased risk for type 2 diabetes.

The current form of the Diabetes Population Risk Tool (DPoRT) includes a non-specific

category of ethnicity (white/non-white in concordance with public data available in Canada).

Having to include detailed ethnic information would limit the applicability of the risk tool since

this information is not routinely collected at the population level. Given the importance of

ethnicity in influencing diabetes risk and its significance in Canada‘s multi-ethnic population, it

is prudent to determine whether detailed ethnic information would significantly improve the

prediction of diabetes risk using a population-based risk tool.

Objective: To apply and compare the Diabetes Population Risk Tool (DPoRT) with a modified

version that includes detailed ethnic information in Canada‘s largest and most ethnically diverse

province, Ontario.

Methods: Two diabetes prediction models were built using the same principles and data as

DPoRT. The 2 models created in this study were: (i) a model that contained predictors specific to

the following ethnic groups: White, Black, Asian, South Asian, and First Nation and (ii) a

reference model which did not include a term for ethnicity. In addition to discrimination and

calibration measures of model performance, 10-year diabetes incidence rate and predicted

number of new diabetes cases in Ontario using the different algorithms were compared. The

algorithms were developed using the 1996-7 National Population Health Survey in Ontario (N =

19,861) and validated in the 2000/1 Canadian Community Health Survey in Ontario

(N=26,465).

90
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Results: All non-white ethnicities were associated with higher risk for developing diabetes than

white ethnicity. Among non-white ethnicities South Asian ethnicity had the highest hazard ratio

for diabetes in both males and females. Discrimination and calibration were similar across all

algorithms (0.75 – 0.77). Sufficient calibration (χ2H-L < 20) was maintained in the development

and validation cohort for all models except the detailed ethnicity models for males ( χ2H-L =

33.9). For both males and females, applying DPoRT resulted in the lowest overall ratio between

observed and predicted diabetes risk compared to the other two algorithms. DPoRT appears to

identify more cases at high risk than the other two algorithms in males, whereas in females both

DPoRT and the full ethnicity model identified more high risk cases compared to the algorithm

without ethnic information. Overall across decile of risk the DPoRT and full ethnicity algorithms

were very similar in terms of predictive accuracy and estimated risk in the population.

Conclusions: Although from the individual risk perspective, incorporating information on

ethnicity may be important, when predicting new cases of diabetes at the population level and

accounting for other risk factors, detailed ethnic information did not improve the discrimination

and accuracy of the model or identify significantly more diabetes cases in the population.

91
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.2.2 Introduction

Planning for health care and public health resources needed to address the significant burden of

diabetes (1) is as an important aspect of population health management, which can be informed

by robust prediction tools, such as the Diabetes Population Risk Tool (DPoRT)( Chapter 2). This

tool can aid policy makers, planners, and physicians by providing reliable estimates of the

upcoming diabetes epidemic. In addition, the effectiveness of widespread prevention strategies

can be improved by knowing which groups to target and how extensive a strategy is needed to

stabilize or reduce the number of new cases.

Risk prediction tools for estimating disease risk are common in clinical settings and are

used for clinical decision-making (2). One of the limitations of clinical risk prediction tools for

population prediction is the reliance on physical measurements or special risk questions, such as

fasting blood sugar (3-5) or diabetes family history (6, 7) in the case of diabetes. At the

population level in Canada, these measurements are not easily, accurately, or systematically

captured. Currently, data on the prevalence of risk factors in the population are only collected

through national population health surveys using self-reported measures. One of the key

attributes of DPoRT is its accessibility to a broad audience. This is achieved by using data from

surveys that are publicly available. Detailed ethnic information, though collected, is not publicly

reported. Ethnic information from the surveys are reported publicly as ―white/non-white,‖ and

thus this form for ethnicity was used in DPoRT in order to ensure that the tool can be applied to

public data.

There is growing evidence that certain ethnic groups are at increased risk for developing

type 2 diabetes. Globally, non-European populations have a higher proportional burden of type 2

diabetes compared to the other regions of the world(8). The highest diabetes rates in the world

92
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

are seen in aboriginal population, including those in Australia(9, 10), United States(11, 12), and

Canada(13, 14). In the United States, studies have shown that those of African and Hispanic

decent are at increased risk for developing diabetes compared with non-Hispanic white

Americans (15-17). Throughout the world, those of South Asian decent are another ethnic group

which has been shown to carry an increase burden of type 2 diabetes compared with both non-

white and white ethnicities (18-20). Data from Ontario demonstrate that overall immigrant and

ethnic minority populations suffer from a higher burden of diabetes and its complications (21).

The importance of ethnicity when considering those at high risk for developing diabetes in the

clinical setting has been emphasized through diabetes guidelines that recommend people of

Aboriginal, Hispanic, south Asian, Asian, or African descent should be targeted for screening

(22, 23). Canada‘s immigrant population is largely made up of non-white ethnicities.

Immigrants account for 18-20% of Canada‘s population (24), and this percentage is expected to

increase over time. Estimates of immigrant populations are as high as 50% for major urban

centers such as Toronto.

Though clinically and epidemiologically important risks associated with ethnicity are

apparent, it is not clear how ignoring ethnic-specific predictors will affect a population-based

prediction tool for diabetes. Currently, given that DPoRT performed well in 2 external validation

cohorts, it is assumed that diabetes risk is sufficiently estimated using the current form of the

model. However, given the significance of ethnicity in Canada and its important influence on

diabetes risk, it may be possible that failing to apply ethnic-specific predictors will reduce the

ability to identify high risk groups. In order to have confidence in applying this tool in Canada, it

needs to be determined if the inclusion of detailed ethnic predictors will significantly change the

performance of DPoRT.

93
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

The purpose of this study was to assess the impact of including detailed ethnic information in a

prediction algorithm for diabetes. Specifically this study described the relative benefits to

predictive accuracy and model outputs that are gained with the addition of ethnic specific

predictors to the model. In addition to informing the application of DPoRT, this work also

provides insight into the independent role of ethnicity on diabetes risk once additional risk

factors are considered. No study has previously taken such an approach (i.e. prediction) to

understand the impact of ethnicity on estimating the probability of developing diabetes.

94
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.2.3 Methods

Creation and validation of DPoRT

Development Cohort

The study cohort was derived from 23,403 people from Ontario that responded to the 1996/7

National Population Health Survey (NPHS-ON) conducted by Statistics Canada. In the NPHS,

households were selected though a stratified, multilevel cluster sampling of private residences

using provinces and/or local planning regions as the primary sampling unit. The survey,

conducted via telephone, had an overall 83% response rate and all responses were self-reported

(15). Persons under the age of 20 (n = 2, 407) and those who had previously diagnosed diabetes

or self-reported diabetes were excluded (n = 894). Those who were pregnant at the time of the

survey were also excluded (n = 241), due to the fact that baseline Body Mass Index (BMI) could

not be accurately ascertained, leaving a total of 19,861 individuals (Figure 1). Sixty-six males

were further excluded due to missing baseline BMI resulting in 9,177 males and 10,618 females

in the final cohort.

Validation Cohorts

The DPoRT algorithm was validated in two external cohorts. Further details on the DPoRT

validation are provided in Chapter 2. One validation cohort was used in this study to compare the

performance of the 3 risk functions in an external cohort. The validation study used in this cohort

was derived from the Ontario portion of the 2000/1 Canadian Community Health Survey (CCHS,

Cycle 1.1, N = 37,473), a national telephone survey administered by Statistics Canada. The

target population of the CCHS consisted of persons aged 12 and over resident in private

dwellings in all provinces and territories, excepting those living on Aboriginal reserves, on

95
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Canadian Forces Bases, or in some remote places. The CCHS included the same self-reported

health questions as the NPHS. Like the NPHS, this survey uses a multistage stratified cluster

design and provides cross-sectional data representative of 98% of the Canadian population over

the age of 12 years, and attained an 80% overall response rate (25, 26). After the exclusion

criteria were applied there were 26,465 individuals in the validation cohort.

Identifying respondents who develop diabetes

Survey data from development and validation cohorts were linked to provincial administrative

health care databases that include all persons covered under the government funded universal

health insurance plan. The diabetes status of all respondents in Ontario was established by

linking persons to the Ontario Diabetes Database (ODD). The Ontario Diabetes Database (ODD)

contains all physician diagnosed diabetes patients in Ontario identified since 1991. The database

is created using hospital discharge abstracts and physician service claims. A patient is said to

have physician diagnosed diabetes if he or she meets at least one of the following two criteria:

(a) a hospital admission with a diabetes diagnosis (International Classification of Diseases

Clinical Modification code 250 (ICD9-CM) before 2002 or ICD-10 code E10 – E14 after 2002,

or (b) a physician services claim with a diabetes diagnosis (code 250) followed within two years

by a either physician services claim or a hospital admission with a diabetes diagnosis.

Individuals entered the ODD as incident cases when they were defined as having diabetes

according to the criteria described above. A hospital record with a diagnosis of pregnancy care

or delivery close to a diabetic record (i.e. a gestational admission date between 90 days before

and 120 days after the diabetic record date), were considered to represent gestational diabetes

and so were excluded. The ODD has been validated against primary care health records as an

96
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

accurate measure of incidence and prevalence of diabetes in Ontario (sensitivity of 86%,

specificity of 97%) (27, 28). Information regarding the vital statistics and eligibility for health

care coverage for linked respondents was captured from the Registered Persons Data Base

(RPDB). The algorithm used to create the ODD was applied to Manitoba‘s administrative health

care data to ascertain physician-diagnosed diabetes status in that province. The ODD algorithm is

applied nationally using provincial administrative registries (known as the National Diabetes

Surveillance System (NDSS)) and has been used and validated in several Canadian provinces

(29).

Variable Definitions

Variables used in this study were obtained from responses in the NPHS and CCHS, including:

age, Body Mass Index (BMI), presence of chronic conditions diagnosed by a health professional

(including hypertension and heart disease), ethnicity, immigration status, smoking status, highest

level of achieved education. Body Mass Index (BMI) in kg/ m2 was used as an indicator of

obesity. Derived BMI, calculated by dividing the weight in kilograms by height squared in

meters-squared directly from the NPHS, is only calculated for respondents aged 30 to 64;

therefore, BMI was calculated using weight and height according to derived variable

specification for those who fell outside the age range of 30 – 64 (30). Ethnicity was ascertained

by the question, 'To which ethnic or cultural groups do your ancestors belong?' Classification of

ethnic groups were: White, Black, Asian, South Asian, and First Nation according to Statistics

Canada‘s definition (31). Statistics Canada releases public-use micro data files of the national

health surveys; however, certain variables are suppressed or modified in these files to protect

privacy. In the public-use file ethnicity is only categorized as white or non-white, derived from

97
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

the response to the ethnicity question (20). The shared population health survey files, which are

available at the provincial level, contain more detailed information (including detailed ethnicity).

Access to the shared data files is highly restricted, which is why DPoRT was developed using

variables from the public use file rather than the shared file. In this study the shared file was used

in order to allow for both forms of the variable (white/non-white vs 5 ethnic groups) to be

compared.

Statistical Analysis

Creation of DPoRT

The goal in the creation of DPoRT was to create a risk algorithm that would accurately predict

diabetes risk with high discrimination and calibration using risk factors that are measured

reliably from health survey data. A detailed description of the development of DPoRT can be

found in Chapter 2. Briefly, for each cohort member, the probability of physician-diagnosed

diabetes was assessed from the interview date until censoring for death or end of follow-up using

a Weibull accelerated failure time model Diabetes risk functions were derived separately for

men and non-pregnant women above the age of 20 without a prior diabetes diagnosis. Each

variable was centred on the population mean before inclusion in the model to allow for easier

calibration with other cohorts. This means that when the algorithm is applied, all variables are

centered to the mean variables for the cohort to which it is being applied allowing levels of risk

to be reflective of the average baseline risk in that cohort. Overall risk (predicted probability) of

diabetes for each person was calculated by multiplying the individual‘s risk factor values by the

corresponding regression coefficients, and summing the products (32). The form of the model

was assessed using likelihood ratio tests to compare nested parametric models (33). A plot of

98
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Log(-logS(t))) vs log(t) was inspected for linearity to assess consistency of the survival times

with the Weibull distribution. Cox-Snell residual plots were also constructed to assess the

adequacy of the Weibull distribution assumption.

All estimates (including betas and variance estimates) incorporated bootstrap replicate

survey weights to accurately reflect the demographics of the Canadian population and account

for the survey sampling design based on selection probabilities and post stratification

adjustments. Variance estimates and 95% confidence intervals were calculated using bootstrap

survey weights(34, 35). All statistics were computed using SAS statistical software (version 9.1

SAS Institute Inc, Cary, NC).

Creation of additional models

Two additional models were created in the NPHS-ON development cohort as described above

except the models were modified to either include ethnic-specific predictors or remove any

ethnic predictors; therefore in total 3 prediction models were compared:

(i) DPoRT minus ethnicity – called ―no ethnicity‖

(ii) DPoRT

(iii) DPoRT plus detailed ethnic information – called ―Full ethnicity model‖

In DPoRT (model (ii)) ethnicity is grouped as in the public use file as white/non-white.

In model (iii) ethnicity is broken up into the categories consistent with the diabetes screening

guidelines (36): White, Black, Asian, South Asian, and First Nation.

99
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Comparison across models

The performance of the model was measured by the discrimination and calibration values in the

external validation cohort. Discrimination was measured using a C statistic modified for survival

data developed by Pencina et al. (37), analogous to the area under the ROC curve (38) .

Calibration was assessed using a modified version of the Hosmer-Lemenshow χ2 statistic

developed by Nam (39, 40). This statistic is computed by dividing the validation cohort into

deciles of predicted risk and compared the observed versus predicted risk in each decile using a

chi-square statistic (see appendix 6.1). To mark sufficient calibration χ2 = 20 was used as a

cutoff (p<0.01), consistent with D‘Agostino‘s validation of the Framingham algorithms (39).

Observed versus predicted cases of diabetes were also compared across ethnicities to examine

the concordance across ethnic groups using the three algorithms.

The policy implications of the 3 models were assessed by applying the model to the

2000/1 data to predict 10-year diabetes incidence rates and cases and then to compare these

values between the algorithms. The proportion of the population who were identified as high risk

were also reported and compared across algorithms. In this study the probability of developing

diabetes was stratified into the following categories: <2%, 2-5%, 5-10%, 11-20%, ≥20% where

11-20% and ≥20% were considered high risk.

In order to describe the impact of disagreement between observed and predicted diabetes

risk as a function of the proportion of the population where that disagreement exists, an index

called the Population Disagreement Index (PDI) was developed. PDI was summarized across

ethnic groups and compared between models.

100
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

This index is defined as follows:

Population Disagreement Index (PDI)

𝑛
𝑂𝑖
PDI = × 𝑃𝑝𝑖
𝑖=0 𝑃𝑖

𝑂𝑖
Where 𝑃𝑖 = Ratio between observed: predicted in subgroup i

𝑃𝑝𝑖 = Proportion of the population made up of subgroup i

i = 1 ….. n where n = number of subgroups in the population

0 < PDI <∞

The unweighted ratio between observed and predicted were calculated to demonstrate the

influence of the distribution of the subgroup (i.e. 𝑃𝑝𝑖 ) This was calculated by sampling taking

𝑛
𝑂𝑖
the overall ratio, i.e. .
𝑖=0 𝑃𝑖

101
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.2.4 Results

The observed 10-year diabetes risk (cumulative incidence rate) in the Ontario development

cohort was 8.8% for males and 7.3% for non-pregnant females aged 20 years and older at

baseline. In addition to non-white ethnicity the other attributes in the model which were

previously validated were: BMI, age (and its interactions), hypertension, smoking, heart disease

and immigrant status. The DPoRT development section shows the multivariate-adjusted hazard

ratios for the risk factors in the DPoRT algorithm for males and females.

Ethnicity

Adjusted hazard ratios for the ethnic categories in DPoRT and the full ethnicity algorithm are

shown in Figure 1. Non-white ethnicity has a hazard ratio of 2.14 95% CI (1.74, 2.63) in males

and 1.71 (1.35, 2.16) in females, adjusted for all other variables in the risk algorithm. In the full

ethnicity model, hazard ratios for specific ethnic groups ranged from 1.11 to 3.02 compared to

white ethnicity. All non-white ethnic groups were at higher diabetes risk than white ethnicity.

South Asian ethnicity had the highest hazard ratio for diabetes in both males and females.

102
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 1. Adjusted Hazard Ratios (white ethnicity as reference) and 95% CIs for Ethnic
Variables in DPoRT.

103
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Model Performance

All 3 models showed good discrimination in the development and validation cohorts (C-statistic

ranging from 0.75 – 0.77). The algorithm with white/non-white ethnicity predictors had slightly

higher discrimination versus a model with no ethnicity (i.e. DPoRT versus no ethnicity

algorithm) (table 2). The full ethnicity algorithm achieved the same discrimination as DPoRT for

males and females. Sufficient calibration ( χ2H-L < 20) was maintained in the development and

validation cohort for all models except the detailed ethnicity model for males (χ2H-L = 33.9). All

models had a similar ratio of observed to predicted risk across decile of risk (figure 1a&b). All 3

models under-predicted risk in south Asian males and the largest under-prediction was in the

model without any ethnic information (figure 2a&b). Of the three risk algorithms DPoRT had

the lowest overall average ratio between the observed and predicted (Figure 3). Weighting by

population proportion significantly reduces the overall disagreement in the population due to the

fact that larger disagreement occurs in smaller proportions of the population.

Table 1. C statistics with 95% confidence intervals and calibration χ2 statistics for 3 algorithms.

Males Females
Cohort Development Validation Development Validation
C-Statistic
Model
0.75 0.76 0.77 0.76
No ethnicity (0.74,0.77) (0.75,0.78) (0.75,0.78) (0.74, 078)
0.76 0.77 0.77 0.76
DPoRT (0.74,0.77) (0.75,0.78) (0.75,0.79) (0.74,0.78)
0.76 0.77 0.77 0.76
Full ethnicity (0.74,0.77) (0.76,0.79) (0.76,0.79) (0.74,0.78)
H-L χ2
Model
No ethnicity 3.22 11.38 9.01 6.05
DPoRT 1.54 15.36 7.02 8.00
Full ethnicity 6.27 33.91 7.18 9.11

104
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 1a. Observed versus predicted in the validation cohort by decile of risk using the
algorithm without ethnicity, with ‗white/non-white‘ ethnicity, and with detailed ethnic predictors
for males.

Males

70000
No Ethnicity
60000

50000

40000
Observed
30000
Predicted
20000

10000

0
1 2 3 4 5 6 7 8 9 10
70000
DPoRT
60000

50000

40000
Observed
30000
Predicted
20000

10000

0
1 2 3 4 5 6 7 8 9 10
70000
Full Ethnicity
60000

50000

40000

30000 Observed

20000 Predicted

10000

0
1 2 3 4 105
5 6 7 8 9 10
Deciles of Risk
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 1b. Observed versus predicted in the validation cohort by decile of predicted risk using
the algorithm without ethnicity, with ‗white/non-white‘ ethnicity, and with detailed ethnic
predictors for females.

Females

70000
No Ethnicity
60000

50000

40000
Observed
30000
Predicted
20000

10000

0
1 2 3 4 5 6 7 8 9 10
70000
DPoRT
60000

50000

40000
Observed
30000
Predicted
20000

10000

0
1 2 3 4 5 6 7 8 9 10
70000
Full Ethnicity Females
60000

50000

40000
Observed
30000
Predicted
20000

10000

0 106
1 2 3 4 5 6 7 8 9 10
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 2a. Observed versus predicted 5-year risk and number of cases by ethnicity for males.

Males
15.00 160
No Ethnicity

Thousands

Predicted number of DM cases

5-year DM rate (%)
120
10.00

80

5.00
40

0.00 0
White Black Asian First South Other
Nation Asian

Observed Rate Predicted Rate Observed Cases Predicted Cases

15.00 160
DPoRT

Thousands

Predicted number of DM cases

5-year DM rate (%)

120
10.00

80

5.00
40

0.00 0
White Black Asian First South Other
Nation Asian

Observed Rate Predicted Rate Observed Cases Predicted Cases

15.00 160
Full Ethnicity
Thousands
5-year DM rate (%)

120
10.00

80
Predicted number of DM cases

5.00
40

0.00 0
White Black Asian First South Other
Nation Asian

Observed Rate Predicted Rate Observed Cases Predicted Cases

107
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 2b. Observed versus predicted 5-year risk and number of cases by ethnicity for females.

Females
15.00 160
No Ethnicity

Thousands

Predicted number of DM cases

5-year DM rate (%) 120
10.00

80

5.00
40

0.00 0
White Black Asian First South Other
Nation Asian

Observed Rate Predicted Rate Observed Cases Predicted Cases

15.00 160
DPoRT

Thousands

Predicted number of DM cases

5-year DM rate (%)

120
10.00

80

5.00
40

0.00 0
White Black Asian First South Other
Nation Asian

Observed Rate Predicted Rate Observed Cases Predicted Cases

15.00 160
Full Ethnicity
Thousands

Predicted number of DM cases

5-year DM rate (%)

120
10.00

80

5.00
40

0.00 0
White Black Asian First South Other
Nation Asian

Observed Rate Predicted Rate Observed Cases Predicted Cases

108
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 3. Unweighted average ratio between observed and predicted cases and weighted average
ratio between observed and predicted (i.e. PDI).

Males
2.0

1.89
Average ratio observed:predicted

1.8

1.6

1.4
1.32

1.2
1.17

1.09 1.08
1.03
1.0
A lgorithm DPoRT Full ethnicity No ethnicity DPoRT Full ethnicity No ethnicity
Unweighted Weighted

Females
1.3
1.27
Average ratio observed:predicted

1.2

1.1
1.09
1.08
1.07
1.06

1.0

0.93

0.9
A lgorithm DPoRT Full ethnicity No ethnicity DPoRT Full ethnicity No ethnicity
Unweighted Weighted

109
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Policy Implications – estimates of 10-year population risk in Ontario

In males the overall 10- year predicted risk ranged from 9.85% for the no ethnicity

model, 10.11% in DPoRT and 10.06% in the full ethnicity model. In females average 10 year

predicted risk ranged from 7.83% for the no ethnicity model, 7.95% in DPoRT and 7.97% in the

full ethnicity model. There were 9,660 more predicted cases in males and 5, 013 predicted cases

in females in DPoRT than with the model without ethnicity. In males, 1,409 less cases were

predicted in the full model compared to DPoRT and in females 934 more cases were predicted in

the full model compared with DPoRT (Table 2).

Ethnic specific risk diverged more in males and females when including ethnic specific

terms (Figure 4). The distribution of risk in the population for males and females can be seen in

Figure 5. Overall the distribution of risk (i.e. the proportion of the population belonging to each

risk category) is similar across the algorithms for females and males. Overall, DPoRT appears to

identify more cases at high risk than the other two algorithms in males, whereas in females both

DPoRT and the full ethnicity identify more high risk cases and are not substantially different.

Across decile of risk the number of diabetes cases predicted using DPoRT and full ethnicity

algorithms were very similar in both males and females (Figure 6).

Table 2. 10-year risk and predicted new diabetes cases in Ontario from baseline in the 2001
CCHS.
Males Females
DIABETES Med Rate, DIABETES Med Rate,
rate (%) % # new cases rate (%) % # new cases
No ethnicity 9.85 5.78 386,964 7.83 4.18 321,724
DPoRT 10.11 6.50 396,624 7.95 4.74 326,737
Full
ethnicity 10.06 6.74 395,215 7.97 4.69 327,671

110
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 4. Average 10-year diabetes risk for males and females by ethnicity using 3 algorithms

Males

io
n ian
Nat As
ian k t er ut
h te
ac rs th hi
As Bl Fi O So W
DPoRT Full Ethnicity
20
15
10-year risk (%)

10
5
No Ethnicity
20
15
10
5
n k n er n te
ia ac t io th ia hi
As Bl a O As W
tN ut
h
rs
Fi So
Ethnicity
Panel variable: Algorithm

Females

io
n ian
at As
ian k tN er ut
h
hi
te
ac rs th
As Bl Fi O So W
DPoRT Full Ethnicity
20
15
10-year risk (%)

10
5
No Ethnicity
20
15
10
5
n k n er n te
ia ac io th ia hi
As Bl Nat O As W
t h
rs ut
Fi So
Ethnicity
Panel variable: Algorithm

111
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Figure 5a. Distribution of diabetes risk (left) and number of new cases
by risk group in the population (below) using 3 algorithms for males.

112
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Figure 5b. Distribution of diabetes risk (left) and number of new cases
by risk group in the population (below) using 3 algorithms for females.

113
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Figure 6. Predicted diabetes cases in 10 years by decile of predicted risk for males and females using three algorithms

140000 140000
Males Females

120000 120000
No Ethnicity No Ethnicity
DPoRT DPoRT
Full Full
100000 100000

80000 80000

60000 60000

40000 40000

20000 20000

0 0
1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
Decile of Risk Decile of Risk

114
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.2.5 Discussion

The aim of this study was to assess the impact of including detailed ethnic predictors in a

population-based risk tool for diabetes in Canada. In addition to identifying relative hazards of

developing diabetes by ethnicity, this study provides estimates of the predicted number of cases

in a provincial population by ethnicity for the next 10-year period. Using a population-based

cohort, this study confirmed that those of non-Caucasian descent are at increased risk for

developing diabetes and consistent with previous research, hazard ratios were highest among

South Asians (18). In terms of overall model performance, no additional predictive value was

detected when adding detailed ethnic predictors. At the population level, distribution of risk was

similar across of different risk levels in the population, particularly between DPoRT and the full

ethnicity model. This study suggests that using DPoRT in its current form is sufficient for

accurately predicting diabetes cases in ethnically diverse population similar to Ontario. The

funding that the algorithm to predict diabetes that uses detailed ethnicity did not significantly

differ from one that uses a broad categorization of ethnicity can be driven by two mechanisms

involving statistical prediction and population disagreement influence (PDI).

There are several reasons why a clinically important risk factor may not improve the

performance of a prediction tool. Even though a variable is independently associated with an

outcome, it may not provide incremental improvements in test characteristics which are relevant

for prediction (i.e. discrimination and calibration) in the context of existing predictors. This

phenomenon has also been shown for other clinical predictors and outcomes such as C-reactive

protein for cardiac risk prediction (41). In fact, research has shown that although a battery of

novel risk factors have been developed for the prediction of major coronary heart disease (CHD)

events, these novel factors have been generally unimpressive in their ability to improve CHD

115
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

prediction (42). Furthermore, it has been shown that for a variable to make significant

improvements in discrimination (i.e. improvement in AUC from 0.8 to 0.9) its multivariable

odds ratio must be 6.9 or greater (43) suggesting that in order for detailed ethnicity to improve

the algorithm beyond its current discrimination, the adjusted hazard ratio must be very large in

magnitude. Interestingly, the model without ethnicity was not detectably worse in terms of model

performance such that discrimination and calibration were only marginally decreased compared

to DPoRT. This is likely due to the fact that many of the reasons that ethnicity plays a role in

diabetes risk are related to other factors captured in the model. In particular, socioeconomic

status, obesity (particularly younger onset of obesity), and other lifestyle factors have been

shown to be related to both ethnicity and diabetes risk (18). Most importantly, immigration

status, which is already captured in the model, may explain a significant amount of variability in

diabetes incidence that is associated with ethnicity. The diminishing return on model

performance when adding statistically significant predictors to the model was also noted in the

model building process of DPoRT and was one of the reasons that DPoRT maintained good

discrimination, even with considerable constraints on variable selection.

Population disagreement influence (PDI) is an extension of the idea of population

attributable risk (PAR). PAR describes the impact of a risk factor on population risk as a

function of the prevalence of the exposure and the relative risk of disease (44). In this study the

estimate of relative risk in PAR is translated to the disagreement between observed and predicted

and the prevalence of the exposure is translated to the prevalence of the population where the

disagreement exists. Therefore, PDI described the impact of disagreement between observed and

predicted risk for a risk tool as a function of the proportion of the population where that

disagreement exists. PDI exemplifies how population risk is driven by where the cases lie in the

116
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

population. This means that a large relative discrepancy between observed and predicted that is

concentrated in a subgroup that covers a small proportion of the population will have less impact

on the overall population estimate of diabetes compared with disagreement of the same

magnitude that affects a larger proportion of the population. This finding emphasizes an

important difference between individual and population risk prediction; differences in

individuals or subgroups may be important if the algorithm was to be applied to an individual,

but these differences may not be as critical if applied for population estimates. This also

identifies a potential difference in the way that algorithms must be independently validated,

depending on whether they are intended for use on the individual or in small subpopulations. Of

course, in the same way PAR is affected by the prevalence of the risk factor in the population,

the influence of disagreement within ethnic groups is affected by the ethnic composition of the

population. Ethnic composition can change over time and the impact of this on the validity of the

algorithm should be regularly assessed. The use of prediction tools at the individual level or in

small subpopulations must be independently validated in specific subpopulations and used with

caution where evidence of poor fit is occurring.

The purpose of this study was to examine the impact of ethnicity on population risk

prediction and not to validate it for use within specific ethnic groups. Nonetheless, looking at

performance within ethnic groups provides important information about diabetes risk by

ethnicity. DPoRT performed well in all ethnic groups (especially in females), with the exception

of South Asian males where even with the inclusion of full ethnic information the algorithm

resulted in an under-prediction of diabetes risk. This can indicate that there is an aspect of risk in

this population which is not captured by either the variables in DPoRT or detailed ethnicity.

This result is consistent with emerging evidence about the nature of metabolic risk in South

117
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Asian males. This population has significantly more insulin resistance than Caucasian

populations even in the absence of excessive obesity (18). It has been proposed that the excessive

insulin resistance in Asian Indians could be explained by an abdominal fat distribution which

may be genetically determined (45). A recent study looking at detailed radiographic and

anthropometric measures in Asian and Caucasian men showed that for a given BMI or waist

circumference, South Asian men had approximately 6% higher total body fat than Caucasian

men. Other studies have shown that adjustments for BMI or waist circumferences to define

obesity do not entirely account for possible differences in inherent insulin resistance in the South

Asian population (46). Several physiological mechanisms for this occurrence have been

proposed including that South Asian men have a defect in adipose tissue metabolism, which

occurs independently of obesity or abdominal fat distribution. These abnormalities of adipose

tissue metabolism are concomitant with insulin resistance (47). These studies indicate that there

may be an important aspect of diabetes risk which is not captured by simply including ethnicity

and BMI along with the other predictors of the model. The type of detailed physiological

information which may be needed is not captured at the population level nor is it feasible to

include in a tool such as DPoRT. Regardless, these differences did not affect the performance of

the model and the validity of overall population estimates of diabetes.

Another difficulty in estimating the ethnicity-diabetes risk among males is the possibility of

confounding by physical activity. Immigrant men are more likely to engage in jobs that require

physical activity on a daily basis (48) which has been shown to reduce the risk of developing

diabetes (49). This may explain why the full ethnicity algorithm actually performs worse than

DPoRT for some ethnic groups. Inclusion of full ethnic information may result in over-fitting of

the model. This phenomenon was also seen during DPoRT creation.

118
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Previous studies indicate that weight cut-offs may differ in their associated risk for diabetes

within ethnicity groups and that different cut-offs should be used to identify those at high risk

(50, 51). Our study suggests that as long as additional risk factor differences among ethnic

groups are captured in the prediction algorithm, the difference may not actually be as substantial

as previous noted. This difference may be due to the fact that previous studies did not fully

account for possible confounders including age and additional metabolic disorders (50). This

study examined the interaction between age-specific BMI and ethnicity and found no significant

differences.

There are several limitations to consider when interpreting the results of this study. Firstly,

the minimal difference detected between DPoRT and the full ethnicity algorithm may not be

found in other populations with different ethnic compositions. Secondly, using self-report

measures from the health survey is a limitation which could affect predictive risk accuracy since

these measures may be more subject to reporting error and bias than clinical measures. For self-

reported height and weight, in general there is a high agreement; however, validation studies

show that weight tends to be slightly underestimated and that height may be slightly

overestimated and as a result reported BMI is generally lower than measured BMI (52-54),

which would result in a slight underestimation of predicted risk (Chapter 3.1).

The possibility of misclassification is also possible with the use of self-reported ethnicity,

even though it is the most common measure of acquiring ethnic information in epidemiological

studies (55). Interestingly self-report ethnicity is generally preferred by epidemiologists and

federal agencies, such as the US and Canadian Census and the National Center for Health

Statistics (56). This is due to the fact the self-identification with ethnicity is most important for

studying influences of lifestyle on disease risk. Misclassification due to self-reported ethnicity

119
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

may be more problematic when examining the genetic associations with disease (57). Another

important limitation is the exclusion of an important subpopulation which compromises the

generalizability of this research to all ethnicities in Canada. The cohorts covered in the NPHS

and CCHS exclude those living on Aboriginal reserves. Therefore, estimates for First Nations

people apply only to those living off-reserve and are not intended to represent First Nations on-

reserve. Previous studies show that First Nations are at greater risk for diabetes than other

members of the Canadian population including off-reserve First Nations counterparts (13, 14, 58-

60). This is an important component of the population to consider for diabetes prevention and a

population risk algorithm developed specifically for on-reserve populations would be beneficial

to estimating overall diabetes burden in Canada.

Overall there are several key messages to be taken from the results of this study. Firstly,

this analysis provides adjusted hazard ratios and risk estimates to quantify the impact of ethnicity

on diabetes risk using a prospective population-based cohort study in Ontario. This is the first

study that reports 10-year risk and number of cases of diabetes from a prediction model

according to ethnicity. These estimates provide key information for predicting diabetes risk at the

provincial or national level, particularly in the increasingly multiethnic Canadian population.

Secondly, this study shows that DPoRT in its current form is as effective or in some cases better

than the algorithm with full ethnic information for predicting diabetes risk at the population

level. Furthermore, it also appears to work well within ethnic groups, in particular for women.

Though overall model performance was good, analysis by ethnicity shows that further research is

required to improve model fit in South Asian males. This study, consistent with other prediction

tools, has reaffirmed that using a measure that has a statistically significant association with a

disease is not enough to improve predictive performance of a model (61).

120
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

3.2.6 References

1. Wild S et al. Global prevalence of diabetes - Estimates for the year 2000 and
projections for 2030. Diabetes Care 2004;27:1047-53.

2. Anderson KM et al. An Updated Coronary Risk Profile - A Statement for Health-

Professionals. Circulation 1991;83:356-62.

3. Eddy DM, Schlessinger L. Validation of the archimedes diabetes model. Diabetes

Care 2003;26:3102-10.

4. Hanley AJG et al. Prediction of type 2 diabetes using simple measures of insulin
resistance - Combined results from the San Antonio Heart Study, the Mexico City
Diabetes Study, and the Insulin Resistance Atherosclerosis Study. Diabetes 2003;52:463-
9.

5. Ito C et al. Prediction of diabetes mellitus (NIDDM). Diabetes Research and

Clinical Practice 1996;34:S7-S11.

6. Herman WH et al. A new and simple questionnaire to identify people at increased

risk for undiagnosed diabetes. Diabetes Care 1995;18:382-7.

7. Lindstrom J, Tuomilehto J. The diabetes risk score: a practical tool to predict type
2 diabetes risk. Diabetes Care 2007;26:725-31.

8. World Health Organization. Report of a WHO consultation on obesity, Obesity:

preventing and managing the global epidemic. 1998. Geneva, World Health
Organization.

9. Odea K. Westernization, Insulin Resistance and Diabetes in Australian

Aborigines. Medical Journal of Australia 1991;155:258-64.

10. Odea K et al. Obesity, Diabetes, and Hyperlipidemia in A Central Australian

Aboriginal Community with A Long History of Acculturation. Diabetes Care
1993;16:1004-10.

11. Mokdad AH et al. The continuing epidemics of obesity and Diabetes in the United
States. JAMA: The Journal of the American Medical Association 2001;286:1195-200.

12. Pavkov ME et al. Changing patterns of type 2 diabetes incidence among Pima
Indians. Diabetes Care 2007;30:1758-63.

13. Harris SB et al. The prevalence of NIDDM and associated risk factors in native
Canadians. Diabetes Care 1997;20:185-7.

121
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

14. Young TK et al. Type 2 diabetes mellitus in Canada's First Nations: status of an
epidemic in progress. Canadian Medical Association Journal 2000;163:561-6.

15. Narayan KMV et al. Lifetime risk for diabetes mellitus in the United States.
JAMA 2003;290:1884-90.

16. Winkleby MA et al. Ethnic and socioeconomic differences in cardiovascular

disease risk factors - Findings for women from the third national health and nutrition
examination survey, 1988-1994. Jama-Journal of the American Medical Association
1998;280:356-62.

17. Brancati FL et al. Diabetes mellitus, race, and socioeconomic status - A

population based study. Annals of Epidemiology 1996;6:67-73.

18. Abate N, Chandalia M. Ethnicity and type 2 diabetes - Focus on Asian Indians.
Journal of Diabetes and Its Complications 2001;15:320-7.

19. Ramachandran A et al. Risk of noninsulin dependent diabetes mellitus conferred

by obesity and central adiposity in different ethnic groups: A comparative analysis
between Asian Indians, Mexican Americans and Whites. Diabetes Research and Clinical
Practice 1997;36:121-5.

20. Ramachandran A et al. Rising prevalence of NIDDM in an urban population in

India. Diabetologia 1997;40:232-7.

21. Manuel D, Schultz S. Diabetes in Ontario: An ICES Practice Atlas. Hux, J. E.,
Booth, G. L., Slaughter, P. M., and Laupacis, A. 4.77-4.94. 2003. Toronto, Institute for
Clinical and Evaluative Sciences.

22. Calonge N et al. Screening for type 2 diabetes mellitus in adults: U.S. Preventive
Services Task Force recommendation statement. Annals of Internal Medicine
2008;148:846-U63.

23. Norris SL et al. Screening adults for type 2 diabetes: A review of the evidence for
the U.S. Preventive Services Task Force. Annals of Internal Medicine 2008;148:855-
U82.

24. Newbold KB, Danforth J. Health status and Canada's immigrant population.
Social Science & Medicine 2003;57:1981-95.

25. Statistics Canada. Canadian Community Health Survey Methodological

Overview. Health Reports 2002;13:9-14.

26. Canadian Community Health Survey 2000–2001. Ottawa: 2003.

27. Hux JE, Ivis F. Diabetes in Ontario. Diabetes Care 2005;25:512-6.

122
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

28. Lipscombe LL, Hux JE. Trends in diabetes prevalence, incidence, and mortality
in Ontario, Canada 1995-2005: a population-based study. Lancet 2007;369:750-6.

29. Health Canada. Responding to the Challenge of Diabetes in Canada. 2003.

Ottawa, ON.

30. Statistics Canada. 1996-7 National Population Health Survey: Derived Variable
Specifications. 1999. Ottawa.

31. Statistics Canada. Population and Dwelling Counts, for Census Divisions, Census
Subdivisions (Municipalities) and Designated Places, 2001 and 1996. 2001.

32. Odell PM, Anderson KM, Kannel WB. New Models for Predicting
Cardiovascular Events. Journal of Clinical Epidemiology 1994;47:583-92.

33. Farewell VT, Prentice RL. A study of distributional shape in life testing.
Technometrics 1977;19:69-75.

34. Yeo D, Mantel H, Lui TP. Bootstrap variance estimation for the National
Population Health Survey. 778-783. 1999. Baltimore, American Statistical Association.

35. Kovacevic MS, Mach L, Roberts G. Bootstrap variance estimation for predicted
individual and population-average risks. Proceedings of the Survey Research Methods
Section. 2008. American Statistical Association.

36. Berg AO et al. Screening for type 2 diabetes mellitus in adults: Recommendations
and rationale. Annals of Internal Medicine 2003;138:212-4.

37. Pencina M, D'Agostino RB. Overall C as a measure of discrimination in survival

analysis: model specific population value and confidence interval estimation. Statistics in
Medicine 2004;23:2109-23.

38. Campbell G. General Methodology I: Advances in statistic methodology for the

evaluation of diagnostic and laboratory tests. Statistics in Medicine 2004;13:499-508.

39. D'Agostino RB et al. Validation of the Framingham Coronary Disease Prediction

Scores. JAMA 2001;286:180-7.

40. Nam B-H. Discrimination and Calibration in Survival Analysis: Extension of the
ROC Curve for Descrimination and Chi-square test for Calibration. 2000. Boston
University.

41. Lloyd-Jones DM et al. Narrative review: Assessment of C-reactive protein in risk

prediction for cardiovascular disease. Annals of Internal Medicine 2006;145:35-42.

42. Wilson PWF et al. C-reactive protein and risk of cardiovascular disease in men
and women from the Framingham Heart Study. Archives of Internal Medicine
2005;165:2473-8.
123
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

43. Pepe MS et al. Limitations of the odds ratio in gauging the performance of a
diagnostic, prognostic, or screening marker. American Journal of Epidemiology
2004;159:882-90.

44. Levin ML, Bertell R. Re - Simple Estimation of Population Attributable Risk

from Case-Control Studies. American Journal of Epidemiology 1978;108:78-9.

45. Banerji MA et al. Body composition, visceral fat, leptin, and insulin resistance in
Asian Indian men. Journal of Clinical Endocrinology and Metabolism 1999;84:137-44.

46. Chandalia M et al. Relationship between generalized and upper body obesity to
insulin resistance in Asian Indian men. Journal of Clinical Endocrinology and
Metabolism 1999;84:2329-35.

47. Abate N et al. Adipose tissue metabolites and insulin resistance in nondiabetic
Asian Indian men. Journal of Clinical Endocrinology and Metabolism 2004;89:2750-5.

48. Norman A et al. Total physical activity in relation to age, body mass, health and
other factors in a cohort of Swedish men. International Journal of Obesity 2002;26:670-5.

49. Qi L, Hu FB, Hu G. Genes, environment, and interactions in prevention of type 2

diabetes: A focus on physical activity and lifestyle changes. Current Molecular Medicine
2008;8:519-32.

50. Barba C et al. Appropriate body-mass index for Asian populations and its
implications for policy and intervention strategies. Lancet 2004;363:157-63.

51. Diaz VA et al. How does ethnicity affect the association between obesity and
diabetes? Diabetic Medicine 2007;24:1199-204.

52. Nawaz H et al. Self-reported weight and height: implications for obesity research.
Journal of Preventive Medicine 2001;20:294-8.

53. Rowland M. Self-reported height and weight. American Journal of Clinical

Nutrition 2007;52:1125-33.

54. Shields M, Gorber SC, Tremblay MS. Estimates of obesity based on self-report
versus direct measures. Health Reports 2008;19:1-16.

55. Comstock RD, Castillo EM, Lindsay SP. Four-year review of the use of race and
ethnicity in epidemiologic and public health research. American Journal of Epidemiology
2004;159:611-9.

56. Gomez SL et al. Inconsistencies between self-reported ethnicity and ethnicity

recorded in a health maintenance organization. Annals of Epidemiology 2005;15:71-9.

57. Burchard EG et al. The importance of race and ethnic background in biomedical
research and clinical practice. New England Journal of Medicine 2003;348:1170-5.
124
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

58. Green C et al. The epidemiology of diabetes in the Manitoba-registered first

nation population - Current patterns and comparative trends. Diabetes Care
2003;26:1993-8.

59. Horn OK et al. Incidence and prevalence of type 2 diabetes in the first nation
community of Kahnawa : ke, Quebec, Canada, 1986-2003. Canadian Journal of Public
Health-Revue Canadienne de Sante Publique 2007;98:438-43.

60. Kaler SN et al. High rates of the metabolic syndrome in a first nations community
in western Canada: Prevalence and determinants in adults and children. International
Journal of Circumpolar Health 2006;65:389-402.

61. Pencina MJ et al. Evaluating the added predictive ability of a new marker: From
area under the ROC curve to reclassification and beyond. Statistics in Medicine
2008;27:157-72.

125
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

4. Determining predictors of body mass index and change in body mass index from 1994 –
2004 using a multilevel growth model

4.1 Abstract

Abstract

Background: Increases in obesity and weight are major contributors to chronic disease around

the world, particularly for type II diabetes. Understanding determinants and trajectories of

weight change is an important aspect of public health prevention of chronic disease through

obesity reduction. Few studies have examined the change in Body Mass Index (BMI) over time

and its association with lifestyle and demographic factors using longitudinal population-based

data.

Objective: To understand the predictors of BMI and BMI trajectories in the Canadian population

over a 10-year period.

Methods: This study uses a population-based sample of 14, 123 adults in cycles 1 – 6 of the

longitudinal National Population Health Survey. BMI (at baseline and over time) was modeled

separately for men and women using multilevel growth models with random and fixed effects.

Demographic and lifestyles variables were investigated for their association with these BMI

outcomes, with lifestyle variables modeled in their time-dependent form.

Results: The multilevel analysis showed that age and initial BMI were associated with higher

BMI and increased physical activity, immigrant status, and smoking were negatively associated

with BMI. Those who were older and had higher BMI had significantly lower rates of BMI

increase over time. Female immigrants were less likely to increase BMI over time compared to

non-immigrants. Adjusting for all factors in the model, an important interaction between income

126
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

and sex was found in which, compared to those with higher income, low-income males had

lower BMI whereas the opposite was true for females.

Conclusions: Lifestyle and demographic factors are associated with BMI and BMI change

over time. Longitudinal data and appropriate analytic techniques, such as multilevel growth

models, are crucial to accurately describing these effects.

127
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

4.2 Introduction

There are few risk factors so influential to the development of a disease as increased Body Mass

Index (BMI) on the development of type 2 diabetes (1-5). The past and projected increases in

diabetes incidence throughout the world have been attributed mainly to the increasing incidence

of obesity in the population (6). From a primary prevention perspective, understanding factors

which influence excess weight is integral in implementing and planning effective diabetes

prevention strategies.

There have been several cross-sectional studies that have examined correlates of BMI;

however, longitudinal studies of weight change are less common. Previous analysis of change in

self-reported weight in Canada has been analyzed using ordinary linear regression (OLS) (7). In

this study, Orpana et al. found, on average, a trend of weight gain in the Canadian population

from 1996/7 to 2004/5 and recommend further research to identify the correlates and causes of

this trend. However, as stated by the authors of this study, OLS is not as efficient as other

statistical methods because it does not exploit all the information present in longitudinal data.

Accordingly, this study uses longitudinal data and multilevel growth modeling techniques to

expand on previous research to achieve a better understanding of the determinants of weight and

weight change in the Canadian population.

Hierarchical or multilevel growth models can be used to model individual change over

time in an intuitive and flexible way. These analytic methods extend the concept of multilevel

modeling, taking into account the hierarchical structure of the data, such that individuals (level-

1) are nested into groups (level-2). Multilevel growth models treat level-1 variables as within-

person differences over time and level-2 variables as between-person differences independent of

time (8). In addition to understanding what factors influence BMI at each time point, multilevel

128
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

growth models allows one to study the factors that influence BMI change. Compared to other

models, this technique exploits all of the available data by using variability that exists within-

and between individuals to inform estimates of associations. In doing so, this approach uses

longitudinal data to model the predictors of weight and predictors of weight change distinctly,

yet efficiently, in one model.

Using Canadian population-based longitudinal data, this study employs multilevel growth

models to assess predictors of BMI and BMI change over time.

129
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

4.3 Methods

Study Population

This analysis was done using the longitudinal National Population Health Survey

(NPHS), a nationally representative longitudinal household survey conducted by Statistics

Canada (9). The NPHS longitudinal sample contains 17,276 persons, from all ages, sampled in

1994/1995. These same persons were interviewed at regular cycles (i.e., every two years) and

will continue to be interviewed up to a total period of 18 years, i.e. 10 cycles. This study uses the

first 6 cycles of the survey, up to 2004/5 and covering a 10-year period. Households were

selected though a stratified, multilevel cluster sampling of private residences using local

planning regions as the primary sampling unit, excluding residents of Indian reserves, long-term

care institutions, prisons, remote areas, and Foreign Service personnel. The survey design was a

two-stage probability sample. The overall response rates were cycle 1, 83.6%; cycle 2, 92.8%;

cycle 3, 88.2%; cycle 4, 84.8%; cycle 5, 80.6%; and cycle 6, 77.4%. The cumulative attrition rate

(i.e. i.e. those who did not complete the questionnaire in all 6 cycles) were: cycle 2 9.3%, cycle 3

15.4%, cycle 4 21.3%, cycle 5 27.3 % and cycle 6. 32.7%. The most significant causes of

attrition were inability to trace and refusal. Average non-response for questionnaire items was <

0.1%. Nevertheless, the methods used in this study do not require respondents to have complete

data for all waves of the survey. Population weights to reflect the population characteristics were

computed based on selection probabilities and post stratification adjustments. Subjects with

missing BMI at the first wave of data collection or those under the age of 18 were excluded from

this analysis.

130
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Variables

Self-reported measures of exposures (highest level of education achieved, income, rural status,

chronic conditions, ethnicity, physical activity, and smoking) were obtained from the NPHS, at

every cycle. The outcome, BMI, was based on self-reported height and weight at each cycle.

BMI is the most commonly measured metric of relative weight and is calculated as weight [in

kilograms (kg)] divided by height [in meters (m)] squared (kg/m2). Baseline BMI (BMI at the

first wave of data collection) was included in the model in order to assess its effect on BMI

change. Categories of physical activity were based upon a Physical Activity Index (PAI). The

PAI is based on an individual's leisure time metabolic energy expenditure (EE). EE is calculated

using the frequency and duration of several physical activities, as well as their metabolic (MET)

value. The MET is the energy cost of the activity expressed as kilocalories (kcal) expended per

kg of body weight per hour of activity, doing a physical activity and the number of times and

time spent on each activity. A PAI < 1.5 kcal/kg/day was considered inactive, moderate was

defined by a 1.5 ≤PAI < 3.0 kcal/kg/day, and active was defined as ≥3.0 kcal/kg/day (10).

Income was assessed using income adequacy which is calculated as the dollar distance between

the individual‘s gross household income in the past 12 months and the low-income cut-off

calculated annually to reflect inflation, and adjusted for household size.

Statistical Analysis

Descriptive statistics on baseline characteristics and average BMI at each survey were reported

as means (for continuous variables) and percentages (for categorical variables). As part of the

descriptive and exploratory analysis, BMI change (expressed as a slope) and BMI at the first

wave of data collection (expressed as intercept) were calculated using OLS regression. For each

131
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

individual a regression line was fit using BMI values at each time point. Then the slopes and

intercepts were averaged across individuals. Unadjusted BMI and rate of BMI change were

compared between sociodemographic and lifestyle factors and tested using a t-test or one-way

ANOVA depending on if the variable was categorical or continuous. All descriptive statistics

were calculated using survey weights to accurately reflect the demographics of the Canadian

population and account for the survey sampling design. Significance and variance estimates

were calculated using a bootstrap method (11).

The multilevel growth model was fit on the longitudinal data using PROC MIXED in

SAS (8) including both fixed and random effects. A random effect describes an estimate in the

model that is generated from a subset of all possible subjects in the population; therefore when

estimating a random effect, the variance for that parameter is also estimated. The multilevel

growth model allows the user to examine both within-individual factors and systematic

differences in growth trajectories that occur across groups. The model solves the within- and

between-individual variation in two stages and can include random effects of variation in slopes

and intercepts between individuals (12, 13). This allows the variances of these parameters to be

estimated. The models are described in detail below.

The level-1 model describes within-person variation for BMI. This model can be written as:

(I) BMIij = π0i + π1i(WAVEij) + εij

Where

i = individual and j = measurement occasion or wave (j = 0 to 5 for each wave of data)

π0i is the intercept of individual i‘s BMI trajectory

π1i is the slope or rate of change for the individual i‘s BMI (i.e. average rate of change
across cycles)

132
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

εij is the deviation from the individuals i‘s trajectory from linearity on occasion j (wave).

BMIij is the subject i‘s BMI value at time j (j goes from 0 – 5). Level-1 equations predict the

smallest data-unit, which in this study are BMI-time values. For example, if 10 subjects are

measured 6 times there are 60 level-1 units. Variables that change over time (time-varying

covariates) may also enter into level-1 equations (Figure 1). In this study, the functional form of

the time variable (WAVE) was tested in both its linear and quadratic form to determine which

form was most appropriate for the data.

The level-2 model represents the between-person differences in the change trajectories

(intercept and slope of BMI over time) and time-invariant characteristics of the individual. The

level-1 model only allows people to vary in the values of their individual growth parameters but

the level 2 model ascribes differences in individuals‘ slopes or intercepts such that individuals

who share common predictor values should, on average, vary only according to their individual

change trajectories. The level-2 or between-person model represents the association between a

predictor (shown here as one dichotomous predictor, Xi = 1 or 0) with each subject‘s estimated

initial BMI (i.e. intercept) and rate of change over time (i.e. slope):

π0i = γ00 + γ01Xi + δ0i (Xi is the predictor of initial BMI)

π1i = γ10 + γ11Xi + δ1i (Xi is the predictor of rate of change of BMI)

ζ0i 0 𝜎2 2
𝜎01
where ~𝑁 , 20
ζ1i 0 𝜎10 𝜎12

The γs represent the structural or fixed effects of the model. The fixed effects capture systematic

inter-individual differences such as age, ethnicity, or immigrant status. In the above model:

133
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

γ00 = mean population initial BMI when Xi = 0

γ01 = effect of covariate Xi on initial average BMI

γ10 = mean population rate of change (slope) BMI when Xi = 0

γ10 = effect of covariate x on rate of change (slope) in BMI.

γ00 and γ10 are the level-2 intercepts which represent the mean initial BMI (intercept) and mean

rate of change of BMI (slope) in the population.

Variance components

δ0i and δ1i are level-2 error variables (residuals) which represent the stochastic variation in the

individual growth models allowing each individual‘s growth parameters (slopes and intercepts)

to differ from the population average (See figure 2). In other words, δ0i and δ1i represent the

between-individual stochastic variation for the level-1 predictors (slope and intercept).

Specifically:

δ0i = Difference between population average initial BMI and individual i‘s level of BMI

δ1i = Difference between population average rate of change of BMI and individual i‘s rate of
change of BMI.
δ0i and δ1i represent the unexplained variation in initital BMI and BMI rate of change. Their

variances, defined by the matrix above (σ02 , σ12 & σ012) represent the population variation in the

individual intercept and slope around the mean initial BMI and the mean rate of change as

defined above in the δ0i and δ1i.

σ02 = population variance in initial BMI

134
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

σ12 = population variance in BMI rate of change

σ012 = population covariance between baseline BMI and BMI rate of change

Interpreting these values (σ02 , σ12 & σ012) allow us to comment on how much heterogeneity

exists in the change parameters after accounting for the variables in the model.

Figure 1. Graphical representation of physical activity as a time-varying and time-

invariant predictor.

Moderately Inactive
Active
active
Time-varying
BMI

Time-invariant

1 2 3
Wave

135
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 2. Ten randomly selected growth trajectories from individuals in the cohort to
represent variation in growth trajectories between individuals in the cohort.

136
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

In growth models, level-1 units typically represent time and are nested within individuals.

Therefore, using the earlier example of 10 subjects measured 6 times, we would have 10 level-2

units.

Therefore, our full joint multilevel growth model can be written as:

BMIij = γ00 + γ01Xi + γ10* (WAVEij) + γ11Xi* (WAVEij) + δ1i δ0i + εij

γ01 describes the effect of covariate X on initial BMI

γ11 describes the effect of covariate X on rate of change of BMI.

Multilevel growth modeling makes use of intra-individual variation and inter-individual variation

(thus accounting for clustering within subjects) in an efficient manner since coefficients are

iteratively estimated at both levels. Although attrition can be a significant problem with

longitudinal data, multilevel models are able to accommodate missing or unbalanced data. The

underlying assumption in multilevel growth models is that each individual‘s observed data are a

random sample from their underlying growth trajectory allowing them to be estimated even

without complete data (14). Also, the model is able to handle between-individual variation in

the timing and frequency of measurements because the timing of each measurement occasion is

treated as an explanatory variable in the model; therefore, data from individuals‘ different

measurement patterns (e.g. some of whom may have been measured only once and others at

several irregularly spaced intervals) can be analyzed simultaneously in the same model. Each

BMI-trajectory is a combination of the observed trajectory for the individual based on the

measured waves, and the model-based trajectory based on the predictor variables.

137
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

The following covariance structures were examined and compared to determine the best

fit with the data: Unstructured (UN), Compound Symmetry (CS), Autoregressive(1) (AR1), and

Heterogeneous AR(1) (ARH). The final covariance structure was chosen based on the most

favorable information criteria [Akaike‘s Information Criteria (AIC) and Bayesian Information

Criteria (BIC)], log-likelihood statistics, and number of iterations. Standard diagnostics with

level-1 and level-2 residuals were performed to check model assumptions of normality and

homoschedasicity.

As mentioned in the NPHS survey households were selected though a stratified,

multilevel cluster sampling of private residences using local planning regions as the primary

sampling unit. In order to account for the way that the respondents were sampled normalized

weights that account for selection probabilities were used. Normalized weights represent the

population bootstrap weights provided by Statistics Canada (described in study population

section) divided by the total sample size; accordingly, the sample size is not inflated but the

differential probabilities associated with the survey design are accounted for. All statistical

analyses were carried out using SAS version 9.2 (SAS Institute, Inc, Cary, NC).

138
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

4.4 Results

There were 17, 276 respondents in the first wave of the longitudinal NPHS. Those under the age

of 18 (n = 3,159) and without BMI information in wave 1 or 2 (n = 354) were excluded from the

cohort. The final sample size was 14, 123, or, separated by sex, 7,496 women and 6,627 men

(Figure 3).

Figure 3. Cohort development.

Starting Cohort
N = 17,276

354 excluded
3,159 excluded <
no BMI in wave 1
18 yrs
or 2

N=13,763

FINAL COHORT
N = 7,496 females

N = 6,627 males

Baseline characteristics of the cohort are shown in table 1. Males had an average baseline BMI

of 25.9 kg/m2 and women at 24.7 kg/m2. Male BMI increased on average by 4.4% and female

BMI increased on average by 5.3% over the 10-year period (Figure 4). OLS regression analysis

revealed that baseline BMI and average change in BMI were associated with a number of

139
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

population characteristics (Table 2 & 3). Higher levels of physical activity and smoking status

were associated with lower baseline BMI; however, these factors also tended to be positively

associated with higher rates of BMI increase over time (not statistically significant). Low

income status was negatively associated with baseline BMI in men but the opposite was true for

women. In a cross-sectional analysis at the amid-point of the study (2003) the association

between income quintile and percentage of individuals that reached the obese cutoff (BMI ≥ 30

kg/m2) were not strongly associated in men; however the likelihood of being obese was strongly

inversely related to income in women (Figure 5).

Table 1. Baseline means and proportions (standard deviations) of characteristics at wave 1

(1994-5), by sex.
Males Females
(N = 6,627) (N = 7,496)
Variable
BMI (kg/m2) 25.9 (0.1) 24.7 (0.1)
Age (yrs) 43 (0.2) 45 (0.2)
Immigrant (%) 19.5 (7.3) 20.0 (6.5)
Low Income (%) 13.1 (5.5) 19.0 (6.3)
Current Smoker (%) 32.7 (7.6) 28.4 (6.9)
Education
< Secondary School 24.7 (9.1) 25.4 (8.9)
Secondary School Grad 15.2 (6.3) 17.3 (6.1)
Post-secondary School Grad 60.1 (8.8) 57.3 (8.3)

140
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 4. Mean unadjusted Body Mass Index (BMI) (+/- standard error) across time
[National Population Health Survey (NPHS) 2-year cycles), by sex.

28

Males
27.1
26.9
27
Females
26.6

26.3
BMI (kg/m2)

26.0 26.0
26 25.9
25.8
25.5

25.2
24.9
25 24.7

24
1 2 3 4 5 6
Cycle

141
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Table 2. Average BMI at the first wave of data collection (i.e. baseline BMI) (+/- standard
error) (i.e. intercepts from OLS) and tests for significant differences between cohort
characteristics by sex.

MALES FEMALES
N = 6,627 N = 7,496
Mean (se)* P-value Mean (se) P-value
or difference from or difference from
reference (se) reference (se)
Overall 25.88 (0.07) 24.70 (0.07)
Age
18-30 24.48 (0.12) (ref) 23.18 (0.14) (ref)
31-60 +1.86 (0.14) <0.0001 +1.88 (0.18) <0.0001
≥ 61 + 1.54 (0.20) <0.0001 +2.25 (0.20) <0.0001
Low Income
No 25.93(0.07) (ref) 24.56 (0.07) (ref)
Yes - 0.59 (0.19) 0.0026 +0.57 (0.19) 0.0027
Immigrant
No 25.92 (0.08) (ref) 24.75 (0.07) (ref)
Yes - 0.48 (0.20) 0.0155 -0.22 (0.19) 0.2842
Physical Activity
Low 25.97 (0.09) (ref) 24.97 (0.09) (ref)
Medium -0.27 (0.17) 0.1139 -0.48 (0.18) 0.0058
High -0.38 (0.16) 0.0152 -1.16 (0.19) <0.0001
Education
< Secondary School 26.12 (0.13) (ref) 25.63 (0.15) (ref)
Secondary School Grad -0.15 (0.23) 0.6139 -0.94 (0.25) 0.002
Post-secondary School Grad -0.49 (0.17) 0.0047 -1.35 (0.18) <0.0001
Marital Status
Married 26.44 (0.08) (ref) 25.17 (0.10) (ref)
Common-law -1.22 (0.22) <0.0001 -0.16 (0.31) <0.0001
Single -2.00 (0.21) <0.0001 -2.05 (0.18) <0.0001
Separated or Divorced -0.60 (0.21) 0.0039 -0.06 (0.24) <0.0001
Widowed -0.65 (0.78) 0.4072 +0.15 (0.22) 0.7904

Current Smoker
No 26.03 (0.08) (ref) 24.92 (0.09) (ref)
Yes -0.66 (0.14) <0.0001 -0.76 (0.15) <0.0001
Any chronic condition
Yes 25.60 (0.11) (ref) 23.94 (0.12) (ref)
No +0.42 (0.13) 0015 +1.28 (0.15) <0.0001
se: standard error - all standard errors are computed using survey bootstrap weights

142
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Table 2. Average change in BMI (+/- standard error) (i.e. slopes from OLS) for significant
differences between cohort characteristics by sex.

MALES FEMALES
Mean (se) P-value Mean (se) P-value
or difference from or difference from
reference (se) reference (se)
Overall 0.206 (0.01) 0.221 (0.01)
Age
18-30 0.365 (0.02) (ref) 0.327 (0.02) (ref)
31-60 -0.186 (0.03) <0.0001 -0.108 (0.02) <0.0001
≥ 61 - 0.414 (0.04) <0.0001 -0.420(0.03) <0.0001
Quartile of baseline BMI
Q1 0.339 (0.03) (ref) 0.265 (0.01) (ref)
Q2 -0.109 (0.03) 0.0005 -0.055 (0.02) 0.0150
Q3 -0.186 (0.03) <0.0001 -0.131 (0.02) <0.0001
Q4 -0.272 (0.03) <0.0001 -0.225 (0.03) <0.0001
Low Income
No 0.207 (0.01) (ref) 0.193 (0.01) (ref)
Yes -0.007 (0.03) 0.8375 -0.050 (0.0.03) 0.0463
Immigrant
No 0.191 (0.03) (ref) 0.187 (0.07) (ref)
Yes - 0.058 (0.20) 0.0750 -0.057 (0.19) 0.0198
Physical Activity
Low 0.162 (0.013) (ref) 0.1588 (0.02) (ref)
Medium +0.047 (0.02) 0.044 + 0.038 (0.02) 0.0913
High +0.038 (0.03) 0.145 + 0.155 (0.03) 0.0091
Education
< Secondary School 0.138 (0.02) (ref) 0.063 (0.02) (ref)
Secondary School Grad +0.008 (0.03) 0.7963 + 0.140 (0.03) <0.0001
Post-secondary School Grad +0.070 (0.02) 0.0040 +0.155 (0.02) <0.0001
Marital Status
Married 0.130 (0.01) (ref) 0.152 (0.03) (ref)
Common-law +0.101 (0.03) <0.0001 +0.145 (0.04) 0.0007
Single +0.035 (0.04) 0.0017 +0.153 (0.04) <0.0001
Separated or Divorced +0.130 (0.04) <0.0001 +0.09 (0.03) 0.0232
Widowed -0.017 (0.03) 0.4115 -0.207 (0.03) <0.0001
Current Smoker
No 0.161 (0.02) (ref) 0.151 (0.01) (ref)
Yes +0.059 (0.02) 0.0029 +0.091 (0.02) <0.0001
Any chronic condition
Yes 0.222 (0.01) (ref) 0.2120 (0.01) (ref)
No -0.080 (0.02) 0.0015 -0.062 (0.02) 0.0007
se: standard error - all standard errors are computed using survey bootstrap weights

143
Figure 6. Conceptual diagram of the mulilevel growth model showing variables in the model at each level.
24

22 21.95
21.01
20.7 p < 0.1109
20 19.98 19.76
19.15
18.86
18
17.75
% Obese

16.9
16

14
Males
13.18
Females p < 0.0001
12

10
Lowest Lower-Middle Middle Upper Middle Highest
Income Adequacy Quintile
 Figure 6. Conceptual diagram of the mulilevel growth model.

Age (γ01)
σ02 Baseline BMI (γ02)
Low income(γ03)
Immigrant (γ04) σε2
Ethnicity (γ05)
Intercept
(π0i)

BMI (y)

Wave
(π1i)
Time-varying covariates
Intercept (γ10) (π2i - π8i)
Age (γ11)
Baseline BMI (γ12) Physical Activity
Smoking Status
σ12 Marital Status

The conceptual diagram of the multilevel growth model can be seen in figure 6 and the results

from the model are shown in table 4. All four covariance structures (CS,ARH, AR, and UN)

produced similar results. However, AIC, BIC and log-likelihood statistics showed ARH(1)and

UN produced the best fit. Since iteration time was quicker for UN, this covariance structure was

chosen for subsequent analyses. Quadratic forms of time were considered but found not to be

statistically significant and not to improve the model properties; therefore, the linear form of

time was used in the model. All variables were assessed for their variability over time. Physical

activity, marital status, and smoking status, varied across waves and thus were kept in their time-

varying form. Income adequacy was kept in its time-invariant form due to its minimal variation

over time and to assess its effect in the level 2 equations.

Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

The multilevel growth model shown in table 4 reveals several important predictors for BMI

and its trajectory. The full specification of the multilevel model for males is shown below:

Level 1 model (within individual) - variables in this model are time-varying

BMIij = π0i + π1i(WAVEij) + π2i(Physical Activity -Moderate) + π3i(Physical Activity -Active) +

π4i(Marital Status–common-law) + π5i(Marital Status–single) + π6i(Marital Status–

separated/divorced) + π7i(Marital Status– widowed) + π8i(Current smoker) + εij

Level 2 model (between individual)

Predictors of initial BMI:

π0i = γ00 + γ01(age) + γ02(Baseline BMI) + γ03(Low Income) + γ04(Immigrant) +γ05(Ethnicity -

black) + γ06(Ethnicity –Asian) + γ07(Ethnicity –Aboriginal) + γ08(Ethnicity – South Asian) +

γ09(Ethnicity – other/mixed) + δ0i

Predictors of BMI rate of change:

π1i = γ10 + γ11(age) + γ12(Baseline BMI) + δ1i

146
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Combined Model:

BMIij = γ00 + γ01(age) + γ02(Baseline BMI) + γ03(Low income) + γ04(immigrant) +γ05(Ethnicity

- black) + γ06(Ethnicity –Asian) + γ07(Ethnicity –Aboriginal) + γ08(Ethnicity – South Asian) +

γ09(Ethnicity – other/mixed)

+ γ10(WAVE) + γ11(age)x(WAVE) + γ12(baseline-BMI)x(WAVE) +

π2i(Physical Activity -Moderate) + π3i(Physical Activity -Active) +

π4i(Marital Status–common-law) + π5i(Marital Status–single) + π6i(Marital Status–

separated/divorced) + π7i(Marital Status– widowed) + π8i(Current Smoker) + π9i(Current

Smoker) + π9i(Has a chronic condition) + εij +δ0i + δ1i(WAVE)

Reference groups:

-For marital status ―married‖ is the reference category

-For physical activity ―inactive‖ is the reference category

-For income ―Medium or High income quintile‖ is the reference category

-For ethnicity ―white‖ is the reference category

147
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Table 4 shows the multilevel results from 3 models. The first model (model 1) described the

model without covariates or the effects of time. This model is also known as the unconditional

growth model since the effects of initial BMI and rate of change are is estimated independent of

any terms in the model. The second model (model 2) shows only age and initial BMI and was

used for comparison when building the multivariate model. The third model (model 3) is the

fully adjusted model. Each multilevel model has two intercepts, one for baseline BMI at wave 1

(γ00) and another for subsequent change in BMI over time 1 (γ10). Average BMI and BMI change

in the unconditional growth model was 26.03 km/m2 and 0.2031 kg/m2 for males 24.92 kg/m2

and 0.2245 kg/m2 for females. When each term was added to the second model, results were

compared to the unconditional and conditional growth model and improvement tested using the

information criteria (AIC and BIC) and likelihood ratio statistics.

Controlling for all variables in the model, males have an average increase in BMI of 0.1655

(± 0.0157) kg/m2 and females have an average increase of 0.2216 (± 0.0110) kg/m2. In model 3

in men and women, increasing age and baseline BMI were associated with an increase in BMI;

however those that were older and/or had a higher BMI at baseline increased at a reduced rate.

Prototypical trajectories, based on the multivariate model in females, demonstrating the effects of

age on rates of change of BMI are shown in figure 7.

In the final model, marital status, physical activity, smoking status, and chronic conditions

(for females) were modeled in their time-varying form due to their fluctuations over the 10 year

period. Figure 6 demonstrates the difference between a time-varying and a time-invariant

predictor. Controlling for the effects of time and other variables in the model, increasing levels

of physical activity were associated with lower levels of BMI and the effect was stronger in

148
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

females than males. Smoking was also associated with lower levels of BMI and being married

was associated increased BMI. Having a chronic condition was not associated with BMI or BMI

rate of change in males but was positively associated with average BMI in females. Income,

immigrant status and ethnicity were modeled in their time invariant form. In the multivariate

models, immigrant status was associated with lower BMI; however the effect was stronger in

females. In addition, South Asian ethnicity was associated with increased BMI compared to

white ethnicity females only. Adjusted for all other factors, in men low income was associated

with lower BMI compared to those in the medium or high income category whereas in women,

low income status was associated with increased BMI. When the models were analyzed with

only the subset of individuals with complete, balanced data, the findings did not differ in

direction or statistical significance; however, the magnitude of the associations was slightly

larger.

The random variance components for the intercept and slope of the multivariate models

are also shown in table 4a&b. The random effects for both initial BMI status (σ02) and rate of

change (σ12) were statistically significant (P<0.0001) in all models, indicating significant

variation in baseline levels of BMI and rate of change within the population after controlling for

all variables in the final model. Furthermore, the covariance between baseline BMI and BMI rate

of change and (σ012) was also significant (P<0.0001) meaning that variance in BMI trajectories

depends on initial BMI. The majority of the variation in initial BMI was explained by the fixed

effects added to the model (σ02 decreased from 13.625 to 0.0781 in males and 22.839 to 0.1143

in females). Age and baseline BMI were the main fixed effects that reduced the variance in

initial BMI.

149
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Variation in trajectories stayed relatively stable across models; which is a common

finding in multilevel growth models (14). The variation in initial status was almost double the

magnitude in females versus males indicating that BMI is much more variable in Canadian

women than men.

150
Table 4a. Results of multilevel growth model in males. All variables in the model were centered to their overall mean.

Model 1 Model 2 Model 3

Symbol Term Fixed coefficient p-value Fixed coefficient p-value Fixed coefficient p-value
(standard error) (standard error) (standard error)
INTERCEPT π0i γ00 Intercept 26.034 (0.052) <.0001 25.445 (0.0157) <.0001 25.429 ( 0.0191) <.0001
γ01 Age 0.0009 (0.001) 0.4110 0.0003 (0.0013) <.0001
γ02 BMI 0.9465 (0.004) 0.0031 0.9418 (0.004) 0.8203
γ03 Low Income -0.0881 (0.0440) 0.0455
γ04 Immigrant -0.0950 (0.0454) 0.0366
ref Ethnicity - White (ref) (ref)
γ05 Ethnicity -Black -0.1697 (0.1090) 0.1197
γ06 Ethnicity -Asian -0.09784 (0.0878) 0.2653
γ07 Ethnicity -Aboriginal 0.03259 (0.1070) 0.7607
γ08 Ethnicity -South 0.1575 (0.1726) 0.3613
Asian
γ09 Ethnicity - Other 0.1132 (0.1546) 0.4639
WAVE π1i γ10 Intercept 0.2031 (0.008) <.0001 0.1655 (0.008) <.0001 0.1655 (0.0157) <.0001
γ11 Age -0.009 (0.006) <.0001 -0.01074 (0.0001) <.0001
γ12 BMI -0.014 (0.002) <.0001 -0.0187 (0.002) <.0001
Time-varying
covariates
Physical Activity - (ref)
Inactive (ref)
π2i Physical Activity - -0.0421 (0.024) 0.0806
Moderate
π3i Physical Activity - -0.0895 (0.027) 0.0009
Active
Marital Status - (ref)
Married (ref)
π4i Marital Status - 0.0767 (0.045) 0.0917
Common law
π5i Marital Status - -0.0703 (0.034) 0.0409
Single
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

π6i Marital Status – -0.1934 (0.042) <.0001

Separated or divorced
π7i Marital Status - -0.09815 (0.077) 0.1998
Widowed
π8i Current Smoker -0.2729 (0.0273) <.0001

Estimated random effects

σ02 13.625 (0.2840) <.0001 0.2033 (0.016) <.0001 0.07807 (0.015) <.0001
σ12 0.2005 (0.006) <.0001 0.2058 (0.006) <.0001 0.2141 (0.007) <.0001
σ01 -0.08259 (0.030) 0.0061 0.094 (0.007) <.0001 0.125 (0.007) <.0001
2
σε 1.9922 (0.018) <.0001 1.884 (0.016) <.0001 2.1295 (0.021) <.0001

Goodness of fit
minus2logL 154417 139376 112726
p-value versus model 1 15041 41691
(df =4; p<0.0001) (df =19; p<0.0001)
versus model 2 26650
(df =15; p<0.0001)

AIC (smaller is 154429 139396 112776

better)
AIC versus model 1 yes yes
Improvement
versus model 2 yes
BIC BIC (smaller is better) 154469 139462 112940
Improvement
versus model 1 yes yes
versus model 2 yes

152
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Table 4b. Results of hierarchical linear model fitting in females. All variables in the model were centered to their overall mean.

Model 1 Model 2 Model 3

Symbol Term Fixed coefficient p-value Fixed coefficient p-value Fixed coefficient p-
(standard error) (standard error) (standard error) value
INTERCEPT π0i γ00 Intercept
24.923 (0.063) <.0001 25.389 ( 0.0169) <.0001 25.383 (0.019) <.0001
γ01 Age 0.001825
0.00393 ( 0.0012) 0.0008 (0.00138) 0.1870
γ02 BMI
0.9488 (0.0034) <.0001 0.9461(0.004) <.0001
γ03 Low Income
0.0939 (0.044) 0.0358
γ04 Immigrant
-0.1368 ( 0.0504) 0.0066
ref Ethnicity - White (ref) (ref) 0.0193
γ05 Ethnicity -Black
0.0505 ( 0.1355) 0.7095
γ06 Ethnicity -Asian
-0.0118 ( 0.1018) 0.9075
γ07 Ethnicity -Aboriginal
-0.0050 (0.1972) 0.9797
γ08 Ethnicity -South
Asian 0.1434 ( 0.1106) 0.0195
γ09 Ethnicity - Other
0.2193 (0.1816) 0.2272
γ010 Chronic condition
0.1113 (0.02635) 0.0264
WAVE π1i γ10 Intercept 0.2245 (0.010) <.0001 0.1815 ( 0.0091) 0.2114 (0.011) <.0001
γ11 Age -0.0073 (0.0006) -0.0078(0.0007) <.0001
γ12 BMI -0.0184(0.0018) -0.0227 (0.0021) <.0001
Time-varying
covariates
Physical Activity -
Inactive (ref)
π2i γ20 Physical Activity -
Moderate -0.0246 ( 0.0262) 0.3467
π3i γ30 Physical Activity -
Active -0.1772 ( 0.0321) <.0001
Marital Status -
Married (ref) (ref)
π4i γ40 Marital Status -
-0.0208 (0.054) 0.0193
153
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Common law
π5i γ50 Marital Status -
Single -0.1821 (0.038) 0.6996
π6i γ60 Marital Status –
Separated or divorced -0.2045 ( 0.0425) <.0001
π7i γ70 Marital Status -
Widowed -0.0808 ( 0.0545) <.0001
π8i γ80 Current Smoker
-0.3129 (0.03219) <.0001

Estimated random effects

σ02
σ12
22.839 (0.441) <.0001 0.3026 (0.023) <.0001 0.1143 (0.023) <.0001
σ01
0.2896 (0.008) <.0001 0.3128 (0.001) <.0001 0.3350 (0.010) <.0001
σε2
-0.1850 (0.008) <.0001 0.1994 (0.001) <.0001 0.2124 (0.011) <.0001
2.4579 (0.021) <.0001 2.3499 (0.020) <.0001 2.6239 (0.024) <.0001

Goodness of fit
minus2logL 188503 171270 143960
p-value versus model 1 17233 44913
(df =4; p<0.0001) (df = 20; p <0.0001)
versus model 2 27310
(df = 16; p <0.0001)

AIC (smaller is
better)
AIC versus model 1
Improvement 188515 171290 143973
versus model 2 yes yes
BIC BIC (smaller is better)
Improvement yes

154
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

versus model 1 188556 171358 144167

versus model 2 yes yes

155
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Figure 7. Prototypical plot of BMI change over time for quartiles of baseline age in females
at population average levels for all other characteristics of the cohort.

156
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

4.5 Discussion

This population-based longitudinal study of adults demonstrates how individual characteristics

are related to BMI and BMI change over time. Several demographic and lifestyle factors

significantly influence baseline BMI (intercept) and rate of change (slope) over 10 years.

Though factors which influence BMI and BMI change over time were generally similar between

men and women there was an important difference in the influence of income, namely that low

income status was protective in men and a risk factor in women. Furthermore, all associations

seemed generally stronger in women.

There are several strengths to this study. This sample is population-based with high

participation (response rates ≥ 77%) and representative of the Canadian population (15).

Furthermore, the longitudinal design allowed this study to overcome limitations associated with

cross-sectional designs or single follow-up cohort designs which are unable to consider

trajectories of growth or predictor variables that vary over time. The multilevel growth model

used in this study assessed patterns of growth trajectories, taking into account both within- and

between-individual variation. Other approaches to study change over time, including OLS

regression, are unable to model these within-individual variations since the data are collapsed to

the level-2(16). An additional strength was the use of time-dependent covariates, which has not

been considered in previous studies. This is particularly important to consider with lifestyle

variables that are known to change over time, such as physical activity or smoking status. A

variable that fluctuates over time that is modeled in its time invariant form can lead to the

regression-to-the mean phenomenon. Leading to misinterpretations of the relationships or the

inability to detect a significant effect(17).

157
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Previous analysis of change in self-reported weight in Canada used OLS (7). Orpana et

al. did not examine the effects of the factors investigated in this study; however they did examine

how change in BMI was influenced by sex and age group. Consistent with our study, they found

that weight gain is higher among young and middle aged-adults compared with those greater

than 65, and that females were more likely to increase over time compared to males.

Evidence from cross-sectional studies have generally agreed with the finding that

physical activity is inversely associated with BMI (18-21); however, longitudinal studies have

been less consistent (22-24) though it is unclear if methodological differences, the definition or

measurement of physical activity, or true differences produced these inconsistencies. No

previous study has considered the time-dependent nature of leisure activity, as was done in this

study, and this may have strengthened the ability to find a significant inverse association. The

association between low BMI and smoking is well documented (25-27) and confirmed in this

study, even when considering the time-varying nature of smoking status, which has not been

done in previous studies. Previous studies have also shown, in general, that immigrant status is

associated with decreased BMI (28-31); and though acculturation is thought to lessen this effect,

and even reverse it after a substantial amount of time, that phenomenon was not observed in the

time period of this study. This study revealed that BMI tends to increase with time for younger

people, and even more so for people who start with normal BMI values (< 25 kg.m2) as in

previous studies (7, 16, 32). This emphasizes the need to consider population demographics

when designing weight intervention strategies. Specifically, that obesity prevention is

particularly important in young adults. It is possible that targeting this group will have the most

impact for preventing obesity in the population, since they are at highest risk for weight gain.

158
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

An interesting finding from this study was the opposite association between low income

status and obesity by sex. In their landmark analysis, Sobal and Stunkard (1989) pointed out that

in developed nations SES influences on weight affect men and women differently and is perhaps

related to attitudes and cultural influences (33). It has been shown consistently that adult women

tend to have a strong inverse relationship between SES indicators and obesity; that is, the higher

the SES the less likely the women were to have a higher BMI and/or gain weight (33-36).

Furthermore the prevalence of obesity and being overweight is paradoxically higher in women

with food insecurity(37-41). There are a variety of explanations for these results. Unhealthy

patterns in nutrition (high-density/low nutrient diets) occur in women to a greater extent than

men. Lower income women are less likely to report reducing calories and snacks and limiting

meal proportions to healthy size (42). Emotional eating, described as the practice of consuming

large quantities of food (usually unhealthy food) in response to feelings instead of hunger, is

more likely to affect women versus men (43). Low SES women may be more likely to

experience emotional eating due to feelings of lack of control over their own lives, depression,

emotional stress and poor self-esteem associated with low social class. Women who lack self-

esteem, have high psychological demand and low controls in their occupation are less likely to

exercise and/or have healthy dietary habits (44). A clear demonstration of the SES effects in

women is the contradictory trend of increased obesity among food-insecure women. This is

known as the ‗hunger-obesity paradox‘ in which under-nutrition and obesity co-exist (40). In

periods of food insecurity, women often change their eating patterns to ensure sufficient food for

their children and/or husbands (43). The psychosocial and emotional changes that occur in times

of food restriction result in an increased likelihood to binge eat when food is available and eat

high-calorie foods (one of the key reasons why food restriction is an ineffective method of

159
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

dieting). This phenomenon occurs in a dose-response manner such that the more severe the

food insecurity the more likely the women is to have a binge pattern of eating (39). Men are less

likely to be influenced by emotional eating brought on by psychosocial effects of low SES or

food insecurity. Furthermore, men are not as susceptible to fluctuations in food consumption

during times of food insecurity compared to women (41). An additional difference is the

potential protective effect of low-income labor. Low SES jobs for men tend to be more

physically active than high SES jobs (44). This increased exposure to physical activity burns

calories and is potentially protective of BMI. Our analysis revealed that relationship status had

an impact on future weight, such that those that were married had higher levels of BMI

compared to those who were in other types of relationship or not in a relationship. Previous

research using an individual-level fixed effects model on a national longitudinal survey from the

U.S. also found a significant relationship between marriage and weight gain and proposed

explanations related to shared meals, decreased individual physical activities, and social

obligations (45).

There are several limitations to consider in this study. First, all factors are self-reported

including height and weight, thus possibility subject to measurement error. Non-differential

error would lead to an underestimation of the true associations(46, 47). In general, there is a high

correlation between self-report and measured height and weight (48), however, some studies

have shown that under or over-estimation of height can be related to factors such as sex and

socioeconomic status (49-51). If present, these self-reporting biases may have affected the

direction of association in our study. Recent research into methods to correct for self-reporting

error in these health surveys have recently been introduced. Incorporating these correction

factors is a subject of future research (52). Second, due to the complex nature of the modeling

160
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

process, the models were kept relatively simple to ensure convergence; therefore, additional

variables related to labor, social support, and specific co-morbid conditions were not considered.

Third, additional parameters important to weight gain were not included in the survey‘s data file.

Specifically, there was no detailed information on reproductive factors or dietary habits, which

are a potentially important aspect of weight and weight gain. With respect to the SES

relationship in women, a previous study demonstrated that a large proportion of the gradient was

explained by reproductive factors (44). Not controlling for reproductive factors could cause

significant confounding in women since women of lower SES are more likely to have children

(and more of them), putting them at increased risk for obesity. Consequently, inability to control

for reproductive factors may overestimate SES effects in women. Finally, attrition occurs across

waves and, though this model can handle missing data, the decrease in participants in later waves

may reduce statistical power to detect relationships or higher powers in the functional form for

change.

Preventing obesity in the population is an important strategy to reduce future chronic

disease burden, particularly for diabetes (53). This study describes the lifestyle and demographic

factors associated with BMI and BMI change over time. The study‘s population-based

longitudinal design and multilevel growth model analyses offered a unique opportunity to

investigate new aspects of these relationships. This study shows that weight and weight gain are

influenced by age, sex and baseline BMI. Of the lifestyle variables, physical activity was

protective for BMI, controlling for the effects of time and accounting for the time-dependent

nature of this variable. Furthermore, this study shows differential effects of lifestyle and socio-

demographic factors on BMI by sex. This study shows that young and low-income women are at

higher risk for increased BMI compared to older women and compared to men of the same ages.

161
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Given that young women are particularly at risk of for developing diabetes in the next 10 years

(Rosella et al, under review) particular attention should be paid to this sub-group. Though not

specifically carried out in this study, this model can be explored for its potential as a tool to

predict future BMI in the population, which may be used when generating future estimates of

obesity and/or diabetes.

162
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

4.6 References

1. Colditz G et al. Weight as a risk factor for clinical diabetes in women. American
Journal of Epidemiology 1990;132:501-13.

2. Colditz G et al. Weight gain as a risk factor for clinical diabetes mellitus in
women. Annals of Internal Medicine 1995;122:481-6.

3. Perry IJ et al. Prospective study of risk factors for development of non-insulin

dependent diabetes in middle aged British men. British Medical Journal 1995;310:555-9.

4. Vanderpump MPJ et al. The incidence of diabetes mellitus in an English

community: a 20-year follow-up of the Wickham Survey. Diabetic Medicine
1996;13:741-7.

5. Wilson P et al. Prediction of incident diabetes mellitus in middle-aged adults.

Archives of Internal Medicine 2007;167:1068-74.

6. Mokdad AH et al. The continuing epidemics of obesity and Diabetes in the United
States. JAMA: The Journal of the American Medical Association 2001;286:1195-200.

7. Orpana HM, Tremblay MS, Fines P. Trends in weight change amoung Canadian
adults. Health Reports 2007;18:9-16.

8. Singer JD. Using SAS PROC MIXED to fit multilevel models, hierarchical
models, and individual growth models. Journal of Educational and Behavioral Statistics
1998;23:323-55.

9. Statistics Canada. National population health survey household component cycle

6 (2005-2006) longitudinal documentation. 2006. Ottawa.

Ref Type: Report

10. Statistics Canada. 1996-7 National Population Health Survey: Derived Variable
Specifications. 1999. Ottawa.

11. Yeo D, Mantel H, Lui TP. Bootstrap variance estimation for the National
Population Health Survey. 778-783. 1999. Baltimore, American Statistical Association.

12. Demidenko E, Stukel TA. Efficient estimation of general linear mixed effects
models. Journal of Statistical Planning and Inference 2002;104:197-219.

13. Stukel TA, Demidenko E. Two-stage method of estimation for general linear
growth curve models. Biometrics 1997;53:720-8.

14. Singer JD, Willett JB. Applied Longitudinal Data Anlaysis:Modeling Change and
Event Occurrence. Oxford: Oxford University Press, 2003.

163
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

15. Statistics Canada. NPHS Public Use Microdata Documentation. 1999. Ottawa,
Canada, Statistics Canada.

16. Heo M et al. Hierarchical linea models for the development of growth curves: an
example with body mass index in overweight/obese adults. Statistics in Medicine
2003;22:1911-42.

17. Streiner DL, Norman GR. Measuring Change. Health Measurement Scales. New
York: Oxford University Press, 2003:194-212.

18. Haapanen N et al. Association between leisure time physical activity and 10-year
body mass change among working-aged men and women. International Journal of
Obesity 1997;21:288-96.

19. Hu FB et al. Walking Compared With Vigorous Physical Activity and Risk of
Type 2 Diabetes in Women . JAMA 1999;282:1433-9.

20. Rissanen AM et al. Determinants of Weight-Gain and Overweight in Adult Finns.

European Journal of Clinical Nutrition 1991;45:419-30.

21. Williamson DF et al. Recreational Physical-Activity and 10-Year Weight Change

in A United-States National Cohort. International Journal of Obesity 1993;17:279-86.

22. Gordon-Larsen P et al. Fifteen-year longitudinal strends in walking patterns and

their impact on weight change. American Journal of Clinical Nutrition 2009;89:19-26.

23. Petersen L, Schnohr P, Sorensen TIA. Longitudinal study of the long-term

relation between physical activity and obesity in adults. International Journal of Obesity
2004;28:105-12.

24. Wilsgaard T, Jacobsen BK, Arnesen E. Determining lifestyle correlates of body

mass index using multilevel analyses: The Tromso study, 1979-2001. American Journal
of Epidemiology 2005;162:1179-88.

25. Fogelholm M et al. Predictors of weight change in middle-aged and old men.
Obesity Research 2000;8:367-73.

26. Meltzer AA, Everhart JE. Unintentional Weight-Loss in the United-States.

American Journal of Epidemiology 1995;142:1039-46.

27. Molarius A et al. Smoking and relative body weight: An international perspective
from the WHO MONICA project. Journal of Epidemiology and Community Health
1997;51:252-60.

28. Goel MS et al. Obesity among US immigrant subgroups by duration of residence.

Jama-Journal of the American Medical Association 2004;292:2860-7.

164
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

29. Park Y et al. Place of birth, duration of residence, neighborhood immigrant

composition and body mass index in New York City. International Journal of Behavioral
Nutrition and Physical Activity 2008;5.

30. Sanchez-Vaznaugh EV et al. Differential effect of birthplace and length of

residence on body mass index (BMI) by education, gender and race/ethnicity. Social
Science & Medicine 2008;67:1300-10.

31. Wandell PE et al. Country of birth and body mass index: A national study of
2,000 immigrants in Sweden. European Journal of Epidemiology 2004;19:1005-10.

32. Bjorvell H, Rossner S. A ten year follow-up of weight change in severely obese
subjecrs treated in a behavioural modification program. International Journal of Obesity
1990;14:88.

33. Sobal J, Stunkard AJ. Socioeconomic status and obesity: a review of the
literature. American Psychological Association 1989;105:260-75.

34. Ball K, Crawford D. Socioeconomic status and weight change in adults: a review.
Social Science in Medicine 2005;60:1987-2010.

35. Kuskowska-Wolk A, Bergstrin T. Trends in body mass index and prevalence of

obesity in Swedish women 1980-1989. Journal of Epidemiology and Community Health
1993;47:195-9.

36. Winkleby MA et al. Ethnic and socioeconomic differences in cardiovascular

disease risk factors - Findings for women from the third national health and nutrition
examination survey, 1988-1994. Jama-Journal of the American Medical Association
1998;280:356-62.

37. Adams E et al. Food insecurity is associated with increased risk of obesity in
California women. Journal of Nutrition 2003;133:1070-4.

38. Basiotis P, Lino M. Food insufficiency and prevalence of overweight among adult
women. Family Economics and Nutrition Review 2003;15:55-7.

39. Kendall A, Olson C, Frongillo E. Relationship of hunger and food insecurity to

food availability and consumption. Journal of the American Dietetic Association
1996;96:1019-24.

40. Olson C. Nutrition and health outcomes associated with food insecurity and
hunger. Journal of Nutrition 1999;129:521S-4S.

41. Townsend M et al. Food insecurity is positively related to overweight in women.

Journal of Nutrition 2001;131:1738-45.

165
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

42. Jeffery RW et al. Socioeconomic differences in health behaviors related to obesity

- The Health Worder Project. Internation Journal of Obesity and Related Metabolic
Disorders 1991;15:689-96.

43. Parker S, Keim K. Epic perspectives of body weight in overweight and obese
women with limited income. Journal of Nutrition Education and Behavior 2004;36:282-9.

44. Wamala S, Wolk A, Orth-Gomer K. Determinants of obesity in relation to

socioeconomic status among middle-aged Swedish women. Preventive Medicine
1997;26:734-44.

45. Averett SL, Sikora A, Argys LM. For better or worse: Relationship status and
body mass index. Economics and Human Biology 2008;6:330-49.

46. Greenland S. The effect of misclassification in the presence of covariates.

American Journal of Epidemiology 1980;112:564-9.

47. Rowland M. Self-reported height and weight. American Journal of Clinical

Nutrition 2007;52:1125-33.

48. Nawaz H et al. Self-reported weight and height: implications for obesity research.
Journal of Preventive Medicine 2001;20:294-8.

49. Bostrom G, Diderichsen F. Socioeconomic differentials in misclassification of

height, weight and body mass index based on questionnaire data. International Journal of
Epidemiology 1997;26:860-6.

50. Niedhammer I et al. Validity of self-reported weight and height in the French
GAZEL cohort. International Journal of Obesity 2000;24:1111-8.

51. Wardle K, Johnson F. Sex differences in the association of socioeconomic status

with obesity. Internation Journal of Obesity and Related Metabolic Disorders
2002;26:1144-9.

52. Gorber SC et al. The feasibility of establishing correction factors to adjust self-
reported estimates of obesity. Health Reports 2009;19.

53. Mayor S. International Diabetes Federation consensus on prevention of type 2

diabetes. International Journal of Clinical Practice 2007;61:1773-5.

166
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

5. Thesis Conclusion

Each of the studies presented in this thesis are distinct in their data and methodological approach

but are linked in their common goal to inform population-based risk prediction for diabetes.

Taken together, these studies can inform public health aspects of diabetes & obesity and

epidemiological methods.

In section 2, a novel risk tool - The Diabetes Population Risk Tool (DPoRT) - used to

estimate the incidence of type 2 diabetes that can be applied at the population level using

publicly available data was created. Four important goals were achieved with this work. First an

algorithm to predict the incidence of diabetes with good discrimination and accuracy was

developed. Secondly, an important policy advantage was achieved by building the tool so that it

can be applied to the current risk factor surveillance data (routinely collected survey data) that is

publicly available in Canada. This allows DPoRT to be used by a wide audience of health

planners to accurately estimate diabetes incidence and quantify the impact of interventions.

Thirdly, the vigor of the validation of DPoRT demonstrates a framework, which should be

applied to the validation of other population-based risk algorithms. Finally, the novel application

of a risk algorithm at the population level reveals and important way to understand distribution

of diabetes risk in populations. The subsequent sections of this thesis were built from specific

aspects related to the performance of the risk tool (as affected by measurement) and BMI change

in the population over time.

One of the key aspects of any epidemiological study is the accurate measurement of

exposures and outcomes. To achieve the goals of DPoRT a balance between data availability and

data detail was sought. This choice led to several questions regarding the consequence of public

survey data. In Chapter 3.1, the impact of measurement error (systematic and random) in self-

167
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

reported height and weight on the performance of a diabetes risk prediction tool was studied.

This study demonstrated that systematic and random error decreased the calibration of a

prediction model and only random error reduced the model‘s discrimination. One of the

particularly interesting findings of this study was the finding that of non-differential error can

introduce systematic bias on predicted risk estimates. This study confirms that level of error in

self-reported height and weight in Canada‘s health surveys unlikely to affect the performance or

validation of DPoRT. Importantly, this study provides a framework to quantify the influence of

measurement error on risk prediction using simulation. This work reaffirms that researchers

developing and validating risk tools must be aware of the presence of measurement and its

impact on the performance of their risk tools. Further efforts must be made to understand the

nature of error in self-reporting measurements and ongoing work to improve the quality of

measurements used in risk algorithms will improve model performance. Understanding the

consequence of measurement error on risk prediction is not only important for population risk

tools but can provide further insight for understanding the influence of measurement properties

which can be used to provide evidence for making decisions about data utilization for different

applications in epidemiology.

The results from the investigation into the impact of ethnicity on diabetes risk (3.2) have

several important implications for application of DPoRT in addition to providing insight into the

independent role of ethnicity in the development of diabetes. From this study we find that

although from the individual risk perspective, focusing on different ethnicity may be important,

when predicting new cases of diabetes at the population level and accounting for other risk

factors, detailed ethnic information did not significantly improve DPoRT population estimates of

new diabetes cases. This indicates that for health planning purposes including detailed ethnic

168
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

information may not improve population estimates of future diabetes risk, which has significant

implications on how the tool can be used. It also demonstrates the diminishing return that is

often seen in prediction tools, such that maximum levels of discrimination and goodness of fit

can often be achieved without detailed information on risk factors. While the clinical appeal of a

using detailed ethnicity for estimating diabetes risk is significant, the statistical reality is that

classifying ethnicity in broader categories which are available to the public work as well as a

model which include detailed ethnic information in the Canadian population. Conclusions drawn

from both studies within the measurement section can be used to inform research on the

influence of measurement properties (error and type) on modeling and statistical prediction.

In building the risk tool for diabetes it was demonstrated that that BMI (a relative

measure of weight for height) overwhelmingly influences the predictions for developing diabetes

in the future. For that reason, clarifying determinants of weight and weight change is essential

when developing strategies to prevent or reduce the future diabetes burden. In monitoring trends

over time researchers are often faced with the dilemma of separating trends between individuals

and trends within individuals. Multilevel growth models allow us to model both these aspects

which strengthen the ability to model trends that vary between and within individuals. In

Chapter 4 predictors of weight and weight change were modeled in a longitudinal sample of

Canadians. Specifically importance of age on baseline obesity levels and rate of change are

quantified. The fact that younger individuals are at greatest risk of increasing obesity reinforce

the fact that obesity prevention is most important in younger adults. DPoRT also reinforces that

those with high BMI at lower ages will have a higher probability of developing diabetes than

those that have higher weight at older ages. The fact that factors that influence BMI substantially

depend on gender (particularly income) reveal that interventions for reducing weight may need

169
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

to be considered differentially between genders. The multilevel growth model provides a more

efficient way for predicting weight change over time and can help inform DPoRT and improve

predictive estimates. In addition, weight interventions modeled using DPoRT can be better

informed by the findings of this model by targeting interventions for those at highest risk for

weight gain. Finally, Chapter 4 demonstrates an important use for multilevel growth models in

epidemiology to understand trends of risk factors or diseases that change over time.

There are several follow-up studies which will be conducted based on the findings of this

thesis. Validation of DPoRT will continue as more effective methods to quantify variation in

case ascertainment in different populations are developed. As the surveillance of risk factors

and diabetes improves, DPoRT can be adapted to become even more accurate, while maintaining

its accessibility for decision makers. Future research on measurement error will include

investigation into differential error with respect to disease status or other characteristics of the

population. Though not specifically carried out in this thesis, the multilevel growth model can be

used to predict future BMI in the population, which may be used when generating future

estimates of obesity and/or diabetes for Canada.

Taken together, this thesis represents a body of literature focused on diabetes risk

prediction at the level of populations. Policy makers and health planners can apply these findings

to estimate and plan for the upcoming diabetes epidemics in a country, such as Canada. The

effectiveness of widespread prevention strategies can be improved by knowing which groups to

target and how extensive a strategy is needed to stabilize or reduce the number of new cases.

In summary, population-based prediction tools are an important aspect to epidemiology

and public health. The methods and data used for DPoRT development and validation can

potentially be applied for other diseases in Canada and represent a new way to understand

170
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

population health. The results and methods from investigating measurement and obesity are

important for DPoRT and to further the practice of epidemiology.

171
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

6. Appendix

172
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

6.1 Glossary of frequently used terms and acronyms

Measures of Predictive Accuracy

Discrimination and Calibration

Discrimination: Discrimination is the ability to differentiate between those who are high risk

and those who are low risk – or in this case those who will and will not develop diabetes given a

fixed set of variables.

The ROC curve is a good way to measure discrimination. An ROC curve repeats all possible

pairings of subjects in the sample who exhibit the outcome and do not exhibit the outcome and

calculates the proportion of correct predictions- essentially being and index of resolution of the

model. This proportion under the receiving operator curve is equal to the C statistic which can

be used to assess the degree of discrimination - 1.0 being perfect discrimination and 0.5 being

no discrimination (1-3). A perfect prediction model would perfectly resolve the population into

those who get diabetes and those who do not. Accuracy is unaffected by discrimination,

meaning a model can posse good discrimination yet poor calibration.

Calibration (This concept is also knows as accuracy and/or reliability in the literature):

Calibration is achieved in a prediction model if it is able to predict future risk with accuracy i.e.

if the predicted probabilities closely agree with the observed outcomes. A model that is not

reliable will have significant over- or under- estimation of risk in the overall population and/or

within certain subgroups. A model with good accuracy model will maintain reliability across

various risk groups and other important subpopulations. Accuracy is not an issue if the purpose

of the predicted model is only to rank-order subjects (2).

173
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Calibration in the Framingham prediction models has been assessed statistically by a statistic

developed by D‘Agostino (4). It is calculated by dividing the cohort into deciles of predicted

risk and comparing observed versus predicted risk resulting in a modified version of Hosmer-

Lemenshow χ2. Other measures of assessing accuracy include graphical methods and

correlations or R2 values between observed and predicted estimates (1).

The formula for the Hosmer-Lemenshow χ2 is:

where Ni =total frequency of subjects in the ith group

Oi = total frequency of event outcomes in the ith group

π = average estimated predicted probability of event for ith group

The Hosmer-Lemeshow statistic is then compared to a chi-square distribution with (g-n) degrees

of freedom. As with the other GOF tests, evidence of lack of fit is demonstrates as the chi-

square value increases and the subsequent p-value decreases.

174
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Reference List

(1) Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: Issues in developing
models, evaluating assumptions and adequacy, and measuring and reducing errors.
Statistics in Medicine 1996; 15:361-387.

(2) Harrell FE. Regression Modeling Strategies With Applications to Linear Models, Logistic
Regression, and Survival Analysis. New York: Springer, 2001.

(3) Pencina M, D'Agostino RB. Overall C as a measure of discrimination in survival analysis:

model specific population value and confidence interval estimation. Statistics in Medicine
2004; 23:2109-2123.

(4) D'Agostino RB, Grundy S, Sullivan LM, Wilson P. Validation of the Framingham
Coronary Disease Prediction Scores. JAMA 2001; 286(2):180-187.

Acronyms

DPoRT: Diabetes Population Risk Tool

NPHS: National Population Health Survey

CCHS: Canadian Community Health Survey

ODD: Ontario Diabetes Database

RPDB: Registered Persons Database

NDSS: National Diabetes Surveillance System (for Canada)

BMI: Body Mass Index (kg/m2)

METS: Metabolic Equivalents (expressing the energy cost of physical activities).

175
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

6.2 Ethics

For the DPoRT work approval to link survey data (NPHS 1996/7, CCHS 2000/1) to

hospital administrative data held at ICES was received from Sunnybrook Health Sciences

Centre Research Ethics Board in January 2005. The study design and use of hospital

administrative data meets the requirements of the ICES research agreement with the

Ontario Ministry of Health and Long Term Care. Ethics approval was sought from the

University of Toronto Research Ethics Board and Sunnybrook and Women‘s College

Health Sciences Centre Research Ethics Board and granted in January 2005. Use of the

Manitoba data was approved by the Health Information Privacy Committee and University

of Manitoba Health REB in January of 2005. Access to the National Population

Longitudinal Health Survey at the Research Data Centre and the University of Toronto was

received by Statistic Canada in June of 2008.

176
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

6.3 Simulation flow chart and SAS code

Study flow chart for simulation

1. Input start values for

simulation 2. Apply variance
3. Simulate ~10,000 BMI
equations (IV) for
•taken from actual data values accorgind to input
random error and (V) for
•varying ICC and bias paramaters, 500 times
bias

6. Calculate predicted 5. Predict probability of 4. Predict probability of

risk , H-L statistic, and C- diabetes according to diabetes according to
statistic for 'true' BMI prediction algorithm prediction algorithm
and observed BMI using observed BMI using 'true' BMI

177
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

SAS code
***** INPUT PARAMETERS *****;
%LET TOTREP = 500; %LET N = 10618;
%LET BETA1 = 0.4565 ; %LET SEED1 = 55 ;
%LET MEANHEIGHT = 1.627; %LET MEANWEIGHT = 64.761;
%LET STDHEIGHT_OBS = 0.069 ;%LET STDWEIGHT_OBS = 12.320 ;
%LET SEEDU = 8971 ;
%LET BETA2 = - 0.00509; %LET PREV = 0.0736;
%let intercept = -10.8967;
%LET SEEDE = 4567; %LET STDBMIe = 5.00 ;
%LET SEED2 = 123 ; %LET SEED3 = 15 ;
%LET SEED4 = 2345 ; %LET SEED5 = 4513;
%LET SEED6 = 9876; %LET CORRHW = 0.311;
%LET ICCHEIGHT = 0.9; %LET ICCWEIGHT = 0.9;
%let b_wt = 0;%let b_ht = 0;

data time;
format start time.;
start = time();
output;
run;

DATA TEMP ;
A1 = &STDHEIGHT_OBS ;
B1 = &STDWEIGHT_OBS * &CORRHW ;
B2 = SQRT(&STDWEIGHT_OBS**2 - B1**2) ;

if &ICCHEIGHT = 1.0 then varht_e = 0;

else if &ICCHEIGHT ne 1.0 then VARHT_E =
&STDHEIGHT_OBS*&STDHEIGHT_OBS*(1-&ICCHEIGHT);
VARHT_TRUE = &STDHEIGHT_OBS*&STDHEIGHT_OBS - VARHT_E;

VAR_OBS_HT = &STDHEIGHT_OBS*&STDHEIGHT_OBS ;
if &ICCwEIGHT = 1.0 then varwt_e = 0;
else if &ICCwEIGHT ne 1.0 then VARWT_E =
&STDWEIGHT_OBS*&STDWEIGHT_OBS*(1-&ICCWEIGHT);
VARWT_TRUE = &STDWEIGHT_OBS*&STDWEIGHT_OBS - VARWT_E;
VAR_OBS_WT = &STDWEIGHT_OBS*&STDWEIGHT_OBS ;

corrtrue = (&corrhw)/sqrt(&iccheight * &iccweight);

A2 = SQRT(VARHT_TRUE);
B3 = SQRT(VARWT_TRUE)* CORRTRUE ;
B4 = SQRT(VARWT_TRUE - B3**2) ;

if &b_wt = 0 then meanweight_true = &MEANWEIGHT;

else if &b_wt ne 0 then meanweight_true = &MEANWEIGHT - &b_wt;

178
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

if &b_ht = 0 then meanheight_true = &MEANhEIGHT;

else if &b_ht ne 0 then meanheight_true = &MEANheIGHT - &b_ht;

RUN;

PROC MEANS DATA=TEMP mean; VAR meanweight_true meanheight_true;

run;

DATA TEMP; SET TEMP;

DO REP = 1 TO &TOTREP ;
DO IND = 1 TO &N ;
Z1 = RANNOR(&SEED1) ;
Z2 = RANNOR(&SEED2) ;
Z3 = RANNOR(&SEED3);
Z4 = RANNOR(&SEED4);
Z5 = RANNOR(&SEED5);
Z6 = RANNOR(&SEED6);
U = RANUNI(&SEEDU) ;
ERROR = RANNOR(&SEEDE);

Y1 = A1 * Z1 ;
Y2 = B1 * Z1 + B2 * Z2 ;

Y3 = A2 * Z1 ;
Y4 = B3 * Z1 + B4 * Z2 ;

HEIGHT_OBS = &MEANHEIGHT + Y1 ;
WEIGHT_OBS = &MEANWEIGHT + Y2 ;

BMI_OBS = WEIGHT_OBS/(HEIGHT_OBS*HEIGHT_OBS);
BMI_OBS_SQ = BMI_OBS*BMI_OBS;

HEIGHT_TRUE = meanheight_true + Y3 ;
WEIGHT_TRUE = meanweight_true + Y4 ;

BMI_TRUE = WEIGHT_TRUE/(HEIGHT_TRUE*HEIGHT_TRUE);
BMI_TRUE_SQ = BMI_TRUE*BMI_TRUE;

output;
end ;
end ;RUN;

DATA TEMP; SET TEMP;

179
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Logit = &intercept + &BETA1 * BMI_TRUE + &BETA2 * BMI_TRUE_SQ ;

elogit = exp(logit) ;
prob = elogit / ( 1 + elogit) ;

LOGIT_obs = &intercept + &BETA1 * BMI_obs + &BETA2 * BMI_obs_SQ ;

elogit_obs = exp(logit_obs) ;
prob_obs = elogit_obs / ( 1 + elogit_obs) ;

DIABETES = 0 ;
IF U LE PROB THEN DIABETES = 1 ;

NEWDIABETES = 1 - DIABETES;

diabetes_obs = 0;
IF U LE PROB_obs THEN DIABETES_obs = 1;
NEWDIABETES_OBS = 1 - DIABETES_OBS;
RUN;

proc sort data=temp; by rep;run;

PROC MEANS DATA=TEMP mean median min max noprint;
VAR PROB prob_obs diabetes diabetes_obs;
by rep;
output out=means mean = probmean probobsmean diabetestrue diabetesobs median = medtrue
medobs min = mintrue minobs
max = maxtrue maxobs ;
RUN;

proc means data=means mean;

var probmean medtrue mintrue maxtrue ;
title1 "Females iccheight = &iccheight iccweight = &iccweight";
title2 "Females weight bias = &b_wt height bias = &b_ht";
run;

ods output BinomialProp = Sens;

ods listing close;
proc freq data=temp (where = (diabetes = 1));
by rep;
tables NEWdiabetes_obs;
exact binomial;
run;
ods listing;

data Sens1; set sens;

if Name1 = '_BIN_';

180
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

Sensitivity = CValue1 + 0;
run;

proc means data=Sens1;

Var Sensitivity;
run;

ods output BinomialProp = SPEC;

ods listing close;
proc freq data=temp (where = (diabetes = 0));
by rep;
tables diabetes_obs;
exact binomial;
run;
ods listing;

data Spec1; set spec;

if Name1 = '_BIN_';
Specificity = CValue1 + 0;
run;

proc means data=Spec1;

Var Specificity;
run;

ods output PearsonCorr = Corr;

ods listing close;
proc corr data=temp;
by rep;
var prob prob_obs;
run;
ods listing;

proc means data=Corr (where = (Variable = 'prob'));

var prob_obs;
run;

/* C stat generation*/

proc rank data=temp out=rankout groups=10;

var prob_obs;

181
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

ranks deciles;
by rep;
run;

proc sort data=rankout; by rep deciles;run;

proc means data=rankout sum noprint;

by rep deciles;
var prob_obs diabetes;
output out=hldiab sum=trueprobsum diabsum;
run;

data hlgof;
set hldiab;
N=_freq_;
chi = (N*(diabsum - trueprobsum)**2)/(trueprobsum*(N-trueprobsum));
run;

proc means data=hlgof sum noprint;

var chi;
by rep;
output out=totalchi sum=chi_total;
run;

data totalchi; set totalchi;

chisqcut = 0; if chi_total ge 20 then chisqcut=1;
pvalue = 1-probchi(chi_total,8);
run;

proc means data=totalchi;var Chi_total pvalue;

run;

proc freq data=totalchi; tables chisqcut;

RUN ;

ods output WilcoxonScores=Wranks;

ods listing close;
proc npar1way data=temp wilcoxan;
class diabetes;
var prob_obs;
by rep;
run;
ods listing;

182
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

data Wranks (keep = class N SumofScores numerator denominator C_stat);set Wranks;

if class = 1;
denominator = N*(&N - N);
numerator = SumofScores - (N*(N+1)/2);
C_stat = numerator/denominator;
by rep;
run;

proc means data=Wranks;

var C_stat;
run;

data time;
set time;
format stop time.;
stop = time();
TIMETORUN = stop - start;
format timetorun mmss.
output;
run;
proc print data=time;
title 'TIME TAKEN TO RUN THE PROGRAM';
RUN;

183
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

6.4 Example of ICC related to BMI distributions

ICC (ρ) is defined as

(II) ρ = σ2true / (σ2true + σ2error)

When no error exists + σ2error = 0 thus ρ = σ2true / (σ2true + 0) = 1.0

The distribution of BMI in our survey data (for females) can be represented below:

10

6
Percent

0
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

BMI_OBS

In the case where ICC = 1.0 (no error in self-reported weight) the true BMI distribution will look

identical:

184
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

ICC = 1.0
10

6
Percent

0
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

BMI_TRUE

However if that observed BMI has 20% of its variability due to random error (eg. Non-

differential reporting error) then the true distribution of BMI will be:

ICC = 0.8
10

6
Percent

0
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

BMI_TRUE

185
Doctor of Philosophy Epidemiology (PhD) Dissertation
Laura C.A. Rosella

And in the case of even more error it would look like as so:

ICC = 0.6
12

10

8
Percent

0
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

BMI_TRUE

Therefore, an incrase in reporting error results in a wider variance and distribution of BMI

observed from the survey.

186
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A.
Rosella

6.5 Full results from simulations

187
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

6.5 a) Results from random error in males

Predicted Probability
Height Weight Mean Mean P- % H-L <
ICC ICC Mean Median Min, Max rHW H-L value 20 rpo O-t Sensitivity Specificity
1.0 1.0 9.31% 8.17% (0.16%, 38.11%) 0.475 9.95 0.3689 97.0% 1.0000 0% 100.00% 100.00%
1.0 0.9 9.18% 8.17% (0.22%, 36.68%) 0.501 11.50 0.2919 92.6% 0.9999 0.13% 99.14% 99.76%
1.0 0.8 9.05% 8.17% (0.32%, 34.92%) 0.531 16.33 0.1397 71.6% 0.9997 0.27% 98.12% 99.52%
1.0 0.7 8.91% 8.17% (0.45%, 32.75%) 0.568 25.21 0.0323 30.8% 0.9991 0.40% 96.94% 99.26%
1.0 0.6 8.76% 8.17% (0.67%, 30.12%) 0.613 40.44 0.0023 3.4% 0.9982 0.55% 95.49% 98.97%
1.0 0.5 8.62% 8.17% (0.10%, 26.85%) 0.672 65.12 0.0000 0.0% 0.9965 0.70% 93.70% 98.65%
0.9 1.0 9.26% 8.17% (0.16%, 37.33%) 0.501 10.16 0.3595 96.6% 0.9980 0.05% 98.78% 99.82%
0.9 0.9 9.13% 8.17% (0.23%, 35.77%) 0.528 12.56 0.2519 90.6% 0.9977 0.18% 98.49% 99.64%
0.9 0.8 8.99% 8.17% (0.32%, 33.87%) 0.560 18.52 0.0990 62.8% 0.9971 0.32% 97.62% 99.41%
0.9 0.7 8.85% 8.17% (0.47%, 31.56%) 0.598 28.95 0.0155 18.2% 0.9961 0.46% 96.46% 99.15%
0.9 0.6 8.71% 8.17% (0.69%, 28.75%) 0.646 46.44 0.0006 1.2% 0.9944 0.60% 94.97% 98.86%
0.9 0.5 8.56% 8.18% (1.11%, 25.28%) 0.708 74.17 0.0000 0.0% 0.9915 0.75% 93.11% 98.53%
0.8 1.0 9.21% 8.18% (0.17%, 36.50%) 0.531 10.79 0.3269 95.0% 0.9906 0.10% 97.41% 99.62%
0.8 0.9 9.07% 8.18% (0.23%, 34.80%) 0.560 14.43 0.1875 81.8% 0.9893 0.24% 97.28% 99.46%
0.8 0.8 8.94% 8.18% (0.33%, 32.76%) 0.594 21.79 0.0544 47.0% 0.9873 0.37% 96.57% 99.25%
0.8 0.7 8.79% 8.18% (0.48%, 30.30%) 0.635 34.26 0.0053 9.0% 0.9842 0.52% 95.49% 99.00%
0.8 0.6 8.65% 8.18% (0.72%, 27.32%) 0.685 54.58 0.0001 0.2% 0.9792 0.66% 94.04% 98.71%
0.8 0.5 8.50% 8.18% (1.11%, 23.65%) 0.751 86.65 0.0000 0.0% 0.9703 0.81% 92.10% 98.38%
0.7 1.0 9.15% 8.19% (0.17%, 35.59%) 0.568 12.28 0.2636 90.2% 0.9751 0.16% 95.75% 99.39%
0.7 0.9 9.02% 8.19% (0.24%, 33.76%) 0.598 17.56 0.1137 66.6% 0.9713 0.29% 95.60% 99.24%
0.7 0.8 8.89% 8.19% (0.34%, 31.58%) 0.623 26.98 0.0204 23.6% 0.9660 0.29% 95.03% 99.04%
0.7 0.7 8.74% 8.19% (0.50 %, 28.97% 0.678 42.31 0.0011 2.2% 0.9579 0.57% 94.04% 98.80%
0.7 0.6 8.59% 8.19% (0.75%, 25.82%) 0.733 66.78 0.0000 0.0% 0.9446 0.59% 92.56% 98.52%
0.7 0.5 8.44% 8.19% (1.12%, 21.95%) 0.803 106.77 0.0000 0.0% 0.9187 0.74% 90.48% 98.17%
0.6 1.0 9.01% 8.19% (0.17%, 34.74%) 0.613 15.42 0.1615 78.0% 0.9462 0.30% 93.65% 99.13%
0.6 0.9 8.97% 8.20% (0.24%, 32.68%) 0.646 23.06 0.0423 41.0% 0.9377 0.34% 93.50% 98.98%

188
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

0.6 0.8 8.83% 8.20% (0.35%, 30.36%) 0.685 35.55 0.0037 7.8% 0.9254 0.48% 92.85% 98.78%
0.6 0.7 8.68% 8.20% (0.51%, 27.60%) 0.733 55.64 0.0001 0.2% 0.9066 0.63% 91.93% 98.54%
0.6 0.6 8.53% 8.20% (0.77%, 24.26%) 0.791 87.75 0.0000 0.0% 0.8740 0.78% 90.36% 98.25%
0.6 0.5 8.38% 8.21% (1.24%, 20.15%) 0.867 142.23 0.0000 0.0% 0.8035 0.93% 87.90% 97.88%
0.5 1 9.05% 8.20% (0.17%, 33.63%) 0.672 22.22 0.0518 46.2% 0.8942 0.26% 91.04% 98.81%
0.5 0.9 8.91% 8.20% (0.25%, 31.55%) 0.708 33.670 0.0052 10.4% 0.8761 0.40% 90.77% 98.65%
0.5 0.8 8.77% 8.21% (0.35%, 29.09%) 0.751 51.852 0.0001 0.2% 0.8496 0.54% 90.13% 98.45%
0.5 0.7 8.63% 8.21% (0.52%, 26.14%) 0.803 80.593 0.0000 0.0% 0.8070 0.68% 88.92% 98.20%
0.5 0.6 8.48% 8.21% (0.80%, 22.57%) 0.867 128.457 0.0000 0.0% 0.7273 0.83% 86.96% 97.88%
0.5 0.5 8.33% 8.21% (0.13%, 18.07%) 0.950 225.916 0.0000 0.0% 0.5130 0.99% 83.39% 97.42%

189
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

6.5 b) Results from random error in females

Predicted Probability
Height Weight Mean Mean P- % H-L <
ICC ICC Mean Median Min, Max rHW H-L value 20 rpo O-t Sensitivity Specificity
1.0 1.0 7.20% 5.82% (0.04%, 33.49%) 0.311 10.45 0.3356 3.2% 96.8% 1.0000 0 100%
1.0 0.9 7.07% 5.82% (0.06%, 33.02%) 0.328 11.69 0.2861 7.2% 92.8% 0.9991 0.13% 99.30%
1.0 0.8 6.95% 5.82% (0.83%, 32.38%) 0.348 15.62 0.1561 25.20% 74.8% 0.9996 0.25% 98.46%
1.0 0.7 6.82% 5.82% (0.12%, 31.22%) 0.372 23.00 0.0456 56.6% 43.4% 0.9989 0.38% 97.50%
1.0 0.6 6.68% 5.82% (0.19%, 27.77%) 0.402 34.71 0.0043 92.60% 7.4% 0.9977 0.52% 96.33%
1.0 0.5 6.54% 5.82% (0.29%, 27.83%) 0.440 52.83 0.0001 99.8% 0.2% 0.9955 0.66% 94.89%
0.9 1.0 7.16% 5.82% (0.04%, 33.29%) 0.328 10.50 0.3361 3.8% 96.2% 0.9990 0.04% 99.05%
0.9 0.9 7.04% 5.82% (0.06%, 32.70%) 0.346 12.32 0.2617 9.80% 90.2% 0.9991 0.16% 98.96%
0.9 0.8 6.91% 5.82% (0.08%, 31.84%) 0.367 16.94 0.1273 30.40% 69.6% 0.9991 0.29% 98.29%
0.9 0.7 6.77% 5.82% (0.13%, 30.64%) 0.382 25.05 0.0291 67.00% 33.0% 0.9989 0.43% 97.36%
0.9 0.6 6.63% 5.82% (0.19%, 29.04%) 0.423 37.72 0.0021 97.0% 3.0% 0.9983 0.57% 96.22%
0.9 0.5 6.49% 5.82% (0.30%, 26.94%) 0.464 57.42 0.0000 100.0% 0.0% 0.9972 0.71% 94.75%
0.8 1.0 7.12% 5.82% (0.04%, 33.02%) 0.348 10.83 0.3198 4.8% 95.2% 0.9953 0.08% 97.75%
0.8 0.9 7.00% 5.83% (0.06%, 32.32%) 0.367 13.37 0.2219 13.0% 87.0% 0.9955 0.21% 98.09%
0.8 0.8 6.87% 5.83% (0.08%, 31.33%) 0.389 18.62 0.0941 35.4% 64.6% 0.9955 0.33% 97.68%
0.8 0.7 6.73% 5.83% (0.13%, 29.89%) 0.416 27.72 0.0172 75.4% 24.6% 0.9955 0.47% 96.88%
0.8 0.6 6.59% 5.83% (0.20%, 28.24%) 0.449 41.81 0.0009 98.2% 1.8% 0.9950 0.61% 95.80%
0.8 0.5 6.45% 5.83% (0.31%, 25.99%) 0.492 62.90 0.0000 100.0% 0.0% 0.9942 0.75% 94.38%
0.7 1.0 7.08% 5.83% (0.04%, 32.68%) 0.372 11.48 0.2929 6.80% 93.2% 0.9879 0.12% 96.61%
0.7 0.9 6.95% 5.83% (0.06%, 31.85%) 0.392 19.40 0.1726 19.4% 80.6% 0.9877 0.25% 96.87%
0.7 0.8 6.82% 5.83% (0.09%, 30.73%) 0.416 21.21 0.0563 47.8% 52.2% 0.9873 0.38% 96.65%
0.7 0.7 6.69% 5.83% (0.13%, 29.25%) 0.444 31.47 0.0081 87.2% 12.8% 0.9867 0.51% 96.02%
0.7 0.6 6.55% 5.83% (0.20%, 27.36%) 0.480 47.02 0.0002 99.8% 0.2% 0.9856 0.65% 95.03%
0.7 0.5 6.40% 5.83% (0.32%, 24.95%) 0.526 70.37 0.0000 100.00% 0.0% 0.9839 0.80% 93.65%
0.6 1.0 7.04% 5.83% (0.04%, 32.29%) 0.402 12.80 0.2360 11.00% 89.0% 0.9748 0.16% 95.03%
0.6 0.9 6.91% 5.84% (0.06%, 31.34%) 0.423 17.36 0.1116 29.40% 70.6% 0.9736 0.29% 95.31%

190
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

0.6 0.8 6.78% 5.84% (0.09%, 30.09%) 0.449 24.86 0.0292 68.60% 31.4% 0.9720 0.42% 95.20%
0.6 0.7 6.64% 5.84% (0.13%, 28.48%) 0.480 36.84 0.0026 95.20% 4.8% 0.9698 0.56% 94.72%
0.6 0.6 6.50% 5.84% (0.21%, 26.44%) 0.518 54.59 0.0000 100.00% 0.0% 0.9666 0.70% 93.84%
0.6 0.5 6.36% 5.84% (0.33%, 23.88%) 0.568 80.58 0.0000 100.00% 0.0% 0.9616 0.84% 92.51%
0.5 1 7.00% 5.84% (0.04%, 31.84%) 0.440 15.28 0.1582 20.20% 79.8% 0.9525 0.00% 93.08%
0.5 0.9 6.87% 5.84% (0.06%, 30.77%) 0.464 21.25 0.0547 50.00% 50.0% 0.9494 0.33% 93.36%
0.5 0.8 6.74% 5.84% (0.09%, 29.38%) 0.492 30.81 0.0084 85.2% 14.8% 0.9452 0.46% 93.31%
0.5 0.7 6.60% 5.84% (0.14%, 27.64%) 0.526 45.26 0.0003 98.80% 1.2% 0.9395 0.60% 92.87%
0.5 0.6 6.46% 5.84% (0.21%, 25.46%) 0.568 66.06 0.0000 100.00% 0.0% 0.9313 0.74% 92.07%
0.5 0.5 6.31% 5.84% (0.34%, 22.76%) 0.622 96.18 0.0000 100.00% 0.0% 0.9186 0.89% 90.80%

191
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A.
Rosella

Table 6 c) C-statistics under random error for males and females

MALES FEMALES
C-statistic C-statistic
Height Weight
ICC ICC Mean StDev Min Max Mean StDev Min Max
1.0 1.0 0.6858 0.0082 0.6628 0.7134 0.7179 0.0091 0.6933 0.7424
1.0 0.9 0.6760 0.0091 0.6512 0.7031 0.7097 0.0094 0.6811 0.7358
1.0 0.8 0.6649 0.0094 0.6397 0.6943 0.7002 0.0096 0.6710 0.7277
1.0 0.7 0.6525 0.0097 0.6243 0.6806 0.6894 0.0010 0.6605 0.7202
1.0 0.6 0.6377 0.0099 0.6090 0.6654 0.6770 0.0100 0.6470 0.7035
1.0 0.5 0.6199 0.0102 0.5928 0.6519 0.6625 0.0103 0.6342 0.6920
0.9 1.0 0.6827 0.0090 0.6597 0.7093 0.7158 0.0092 0.6899 0.7401
0.9 0.9 0.6727 0.0092 0.6462 0.6991 0.7074 0.0009 0.6803 0.7316
0.9 0.8 0.6613 0.0095 0.6345 0.6903 0.6978 0.0096 0.6668 0.7271
0.9 0.7 0.6482 0.0097 0.6198 0.6758 0.6868 0.0099 0.6579 0.7166
0.9 0.6 0.6327 0.0100 0.6058 0.6607 0.6743 0.0100 0.6469 0.7033
0.9 0.5 0.6131 0.0102 0.5862 0.6467 0.6592 0.0104 0.6296 0.6869
0.8 1.0 0.6787 0.0091 0.6558 0.7033 0.7131 0.0093 0.6873 0.7381
0.8 0.9 0.6683 0.0094 0.6439 0.6949 0.7044 0.0095 0.6757 0.7315
0.8 0.8 0.6565 0.0096 0.6291 0.6853 0.6949 0.0097 0.6643 0.7246
0.8 0.7 0.6428 0.0098 0.6164 0.6720 0.6836 0.0098 0.6566 0.7139
0.8 0.6 0.6265 0.0101 0.5978 0.6575 0.6706 0.0101 0.6424 0.6989
0.8 0.5 0.6067 0.0101 0.5791 0.6407 0.6553 0.0105 0.6250 0.6841
0.7 1.0 0.6735 0.0092 0.6507 0.7009 0.7097 0.0093 0.6828 0.7353
0.7 0.9 0.6625 0.0095 0.6364 0.6885 0.7009 0.0095 0.6725 0.7291
0.7 0.8 0.6499 0.0096 0.6251 0.6778 0.6909 0.0097 0.6586 0.7220
0.7 0.7 0.6354 0.0098 0.6102 0.6666 0.6795 0.0099 0.6493 0.7090
0.7 0.6 0.6181 0.0101 0.5888 0.6499 0.6661 0.0101 0.6394 0.6980
0.7 0.5 0.5960 0.0102 0.5700 0.6266 0.6502 0.0106 0.6215 0.6795
0.6 1.0 0.6659 0.0094 0.6407 0.6930 0.7053 0.0094 0.6799 0.7339
0.6 0.9 0.6542 0.0095 0.6293 0.6806 0.6960 0.0095 0.6667 0.7242
0.6 0.8 0.6409 0.0098 0.6149 0.6683 0.6858 0.0010 0.6556 0.7157
0.6 0.7 0.6250 0.0099 0.5959 0.6552 0.6741 0.0101 0.6451 0.7017
0.6 0.6 0.6054 0.0102 0.5772 0.6347 0.6601 0.0102 0.6336 0.6913
0.6 0.5 0.5796 0.0104 0.5520 0.6145 0.6435 0.0106 0.6104 0.6719
0.5 1 0.6550 0.0095 0.6279 0.6810 0.6993 0.0095 0.6720 0.7286
0.5 0.9 0.6423 0.0099 0.6148 0.6708 0.6898 0.0097 0.6625 0.7202
0.5 0.8 0.6272 0.0098 0.6014 0.6535 0.6790 0.0099 0.6516 0.7068
0.5 0.7 0.6089 0.0103 0.5824 0.6387 0.6664 0.0102 0.6392 0.6947
0.5 0.6 0.5852 0.0103 0.5591 0.6188 0.6518 0.0103 0.6238 0.6804
0.5 0.5 0.5492 0.0108 0.5193 0.5841 0.6344 0.0107 0.5987 0.6616
192
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Table 6d) Results from bias simulations for males

Height ICC Weight ICC Bias Height Bias Weight Mean o-t Median Min, Max
1.0 1.0 0 0 9.31% 0.00% 8.17% (0.16%, 38.11%)
1.0 1.0 0 Underestimate by 0.5 kg 9.54% -0.23% 8.40% (0.17%, 38.34%)
1.0 1.0 0 Underestimate by 1.0 kg 9.77% -0.46% 8.64% (0.18%, 38.57%)
1.0 1.0 0 Underestimate by 1.5 kg 10.00% -0.69% 8.88% (0.19%, 38.78%)
1.0 1.0 0 Underestimate by 2.0 kg 10.25% -0.94% 9.12% (0.20%, 38.99%)
1.0 1.0 0 Underestimate by 2.5 kg 10.49% -1.18% 9.37% (0.21%, 39.19%)
1.0 1.0 0 Underestimate by 3.0 kg 10.74% -1.43% 9.63% (0.23%, 39.39%)
1.0 1.0 Overestimate by 0.5 cm 0 9.53% -0.22% 8.38% (0.17%, 38.43%)
1.0 1.0 Overestimate by 1.0 cm 0 9.76% -0.45% 8.60% (0.17%, 38.74%)
1.0 1.0 Overestimate by 1.5 cm 0 10.00% -0.69% 8.82% (0.17%, 39.04%)
1.0 1.0 Overestimate by 2.0 cm 0 10.23% -0.92% 9.06% (0.18%, 39.33%)
1.0 1.0 Overestimate by 2.5 cm 0 10.47% -1.16% 9.30% (0.18%, 39.60%)
1.0 1.0 Overestimate by 3.0 cm 0 10.72% -1.41% 9.54% (0.18%, 39.87%)
1.0 1.0 Overestimate by 2.5cm Underestimate by 1.7 kg 10.22% -0.91% 9.09% (0.20%, 39.01%)
1.0 1.0 Overestimate by 2.5 cm Underestimate by 3.2 kg 12.14% -2.83% 11.04% (0.26%, 40.70%)
1.0 1.0 Overestimate by 5.0 cm Underestimate by 1.7 kg 12.70% -3.39% 11.58% (0.25%, 41.28%)
1.0 1.0 Overestimate by 5.0 cm Underestimate by 3.2 kg 13.56% -4.25% 12.50% (0.29%, 41.62%)
0.9 0.9 Overestimate by 2.5cm Underestimate by 1.7 kg 10.04% -0.73% 9.10% (0.28%, 36.85%)
0.9 0.9 Overestimate by 2.5 cm Underestimate by 3.2 kg 11.97% -2.66% 11.04% (0.37%, 39.02%)
0.9 0.9 Overestimate by 5.0 cm Underestimate by 1.7 kg 12.52% -3.21% 11.59% (0.35%, 39.83%)
0.9 0.9 Overestimate by 5.0 cm Underestimate by 3.2 kg 13.39% -4.08% 0.24% (0.41%, 40.36%)
0.8 0.8 Overestimate by 2.5cm Underestimate by 1.7 kg 9.85% -0.54% 9.11% (0.40%, 34.00%)
0.8 0.8 Overestimate by 2.5 cm Underestimate by 3.2 kg 11.77% -2.46% 11.05% (0.53%, 36.56%)
0.8 0.8 Overestimate by 5.0 cm Underestimate by 1.7 kg 12.33% -3.02% 11.60% (0.51%, 37.54%)
0.8 0.8 Overestimate by 5.0 cm Underestimate by 3.2 kg 13.21% -3.90% 12.52% (0.60%, 38.27%)

193
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

Table 6e) Results from bias simulations for females

Height ICC Weight ICC Bias Height Bias Weight Mean o-t Median Min, Max
1.0 1.0 0 0 7.20% 0.00% 5.82% (0.04%, 33.49%)
1.0 1.0 0 Underestimate by 0.5 kg 7.42% -0.22% 6.03% (0.05%, 33.57%)
1.0 1.0 0 Underestimate by 1.0 kg 7.64% -0.44% 6.26% (0.05%, 33.64%)
1.0 1.0 0 Underestimate by 1.5 kg 7.87% -0.67% 6.49% (0.05%, 33.70%)
1.0 1.0 0 Underestimate by 2.0 kg 8.10% -0.90% 6.72% (0.05%, 33.76%)
1.0 1.0 0 Underestimate by 2.5 kg 8.33% -1.13% 6.96% (0.06%, 33.80%)
1.0 1.0 Overestimate by 3.0 cm Underestimate by 3.0 kg 8.57% -1.37% 7.20% (0.06%, 33.84%)
1.0 1.0 Overestimate by 2.5cm 0 7.39% -0.19% 5.99% (0.04%, 33.59%)
1.0 1.0 Overestimate by 2.5 cm 0 7.58% -0.38% 6.17% (0.04%, 33.68%)
1.0 1.0 Overestimate by 5.0 cm 0 7.77% -0.57% 6.35% (0.04%, 33.75%)
1.0 1.0 Overestimate by 5.0 cm 0 7.97% -0.77% 6.54% (0.04%, 33.81%)
1.0 1.0 Overestimate by 2.5cm 0 8.18% -0.98% 6.74% (0.04%, 33.86%)
1.0 1.0 Overestimate by 2.5 cm 0 8.39% -1.19% 6.94% (0.04%, 33.90%)
1.0 1.0 Overestimate by 5.0 cm Underestimate by 1.7 kg 8.06% -0.86% 6.67% (0.05%, 33.76%)
1.0 1.0 Overestimate by 5.0 cm Underestimate by 3.2 kg 9.79% -2.59% 8.42% (0.07%, 34.00%)
1.0 1.0 Overestimate by 2.5cm Underestimate by 1.7 kg 10.18% -2.98% 8.77% (0.07%, 34.04%)
1.0 1.0 Overestimate by 2.5 cm Underestimate by 3.2 kg 11.02% -3.82% 9.69% (0.08%, 34.04%)
0.9 0.9 Overestimate by 5.0 cm Underestimate by 1.7 kg 7.89% -0.69% 6.68% (0.07%, 33.22%)
0.9 0.9 Overestimate by 5.0 cm Underestimate by 3.2 kg 9.63% -2.43% 8.43% (0.10%, 33.86%)
0.9 0.9 Overestimate by 3.0 cm Underestimate by 1.7 kg 10.02% -2.82% 8.78% (0.09%, 33.96%)
0.9 0.9 Overestimate by 2.5cm Underestimate by 3.2 kg 10.87% -3.67% 9.70% (0.11%, 34.00%)
0.8 0.8 Overestimate by 2.5cm Underestimate by 1.7 kg 7.77% -0.57% 6.68% (0.11%, 32.12%)
0.8 0.8 Overestimate by 2.5 cm Underestimate by 3.2 kg 9.47% -2.27% 8.43% (0.15%, 33.37%)
0.8 0.8 Overestimate by 5.0 cm Underestimate by 1.7 kg 9.85% -2.65% 8.79% (0.14%, 33.66%)
0.8 0.8 Overestimate by 5.0 cm Underestimate by 3.2 kg 10.71% -3.51% 9.70% (0.17%, 33.83%)

194
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

6e) H-L and C-statistics from bias simulations for males

Height Mean H- % H-L

ICC Weight ICC Bias Height Bias Weight L Mean P-value <20 rop Sensitivity Specificity
1.0 1.0 0 0 9.951 0.3689 97.0% 1.0000 100.00% 100.00%
1.0 1.0 0 Underestimate by 0.5 kg 10.885 0.3239 95.4% 1.0000 97.60% 100.00%
1.0 1.0 0 Underestimate by 1.0 kg 13.036 0.2340 87.2% 0.9999 95.29% 100.00%
1.0 1.0 0 Underestimate by 1.5 kg 16.536 0.1343 73.0% 0.9998 93.01% 100.00%
1.0 1.0 0 Underestimate by 2.0 kg 21.415 0.0562 49.0% 0.9997 90.86% 100.00%
1.0 1.0 0 Underestimate by 2.5 kg 27.790 0.0190 23.2% 0.9995 88.73% 100.00%
1.0 1.0 0 Underestimate by 3.0 kg 35.653 0.0045 7.4% 0.9993 86.68% 100.00%
1.0 1.0 Overestimate by 3.0 cm 0 10.790 0.3274 95.4% 1.0000 97.67% 100.00%
1.0 1.0 Overestimate by 2.5cm 0 12.801 0.2418 87.4% 1.0000 95.39% 100.00%
1.0 1.0 Overestimate by 2.5 cm 0 16.024 0.1470 75.0% 0.9999 93.16% 100.00%
1.0 1.0 Overestimate by 5.0 cm 0 20.609 0.0677 52.4% 0.9998 91.00% 100.00%
1.0 1.0 Overestimate by 5.0 cm 0 26.615 0.0237 25.8% 0.9997 88.89% 100.00%
1.0 1.0 Overestimate by 2.5cm 0 34.137 0.0058 9.4% 0.9995 86.82% 100.00%
1.0 1.0 Overestimate by 2.5 cm Underestimate by 1.7 kg 20.757 0.0647 51.4% 0.9997 91.09% 100.00%
1.0 1.0 Overestimate by 5.0 cm Underestimate by 3.2 kg 106.266 0.0000 0.0% 0.9977 76.69% 100.00%
1.0 1.0 Overestimate by 5.0 cm Underestimate by 1.7 kg 146.077 0.0000 0.0% 0.9971 73.31% 100.00%
1.0 1.0 Overestimate by 2.5cm Underestimate by 3.2 kg 224.760 0.0000 0.0% 0.9954 68.65% 100.00%
0.9 0.9 Overestimate by 2.5 cm Underestimate by 1.7 kg 21.379 0.0574 50.2% 0.9980 92.46% 99.97%
0.9 0.9 Overestimate by 5.0 cm Underestimate by 3.2 kg 103.347 0.0000 0.0% 0.9969 77.84% 100.00%
0.9 0.9 Overestimate by 5.0 cm Underestimate by 1.7 kg 141.070 0.0000 0.0% 0.9963 74.36% 100.00%
0.9 0.9 Overestimate by 3.0 cm Underestimate by 3.2 kg 220.009 0.0000 0.0% 0.9951 69.53% 100.00%
0.8 0.8 Overestimate by 2.5cm Underestimate by 1.7 kg 29.396 0.0149 23.0% 0.9896 91.86% 99.71%
0.8 0.8 Overestimate by 2.5cm Underestimate by 3.2 kg 109.846 0.0000 0.0% 0.9890 79.01% 100.00%
0.8 0.8 Overestimate by 2.5 cm Underestimate by 1.7 kg 145.911 0.0000 0.0% 0.9882 75.49% 100.00%
0.8 0.8 Overestimate by 5.0 cm Underestimate by 3.2 kg 227.025 0.0000 0.0% 0.9880 70.48% 100.00%

195
Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

6f) H-L from bias simulations for females

Height Mean % H-L

ICC Weight ICC Bias Height Bias Weight H-L Mean P-value <20 rop Sensitivity Specificity
1.0 1.0 0 0 10.455 0.3356 96.8% 1.0000 100.00% 100.00%
1.0 1.0 0 Underestimate by 0.5 kg 11.659 0.2847 93.0% 1.0000 97.11% 100.00%
1.0 1.0 0 Underestimate by 1.0 kg 14.528 0.1922 80.2% 0.9998 94.29% 100.00%
1.0 1.0 0 Underestimate by 1.5 kg 19.354 0.0961 58.6% 0.9997 91.59% 100.00%
1.0 1.0 0 Underestimate by 2.0 kg 25.948 0.0370 32.2% 0.9994 89.00% 100.00%
1.0 1.0 0 Underestimate by 2.5 kg 34.611 0.0090 11.8% 0.9991 86.50% 100.00%
1.0 1.0 0 Underestimate by 3.0 kg 45.477 0.0014 2.0% 0.9986 84.11% 100.00%
1.0 1.0 Overestimate by 3.0 cm 0 11.340 0.2964 93.4% 1.0000 97.51% 100.00%
1.0 1.0 Overestimate by 2.5cm 0 13.343 0.2236 85.6% 0.9999 95.07% 100.00%
1.0 1.0 Overestimate by 2.5 cm 0 16.768 0.1395 69.4% 0.9998 92.68% 100.00%
1.0 1.0 Overestimate by 5.0 cm 0 21.661 0.0667 48.2% 0.9997 90.34% 100.00%
1.0 1.0 Overestimate by 5.0 cm 0 27.848 0.0255 25.8% 0.9995 88.10% 100.00%
1.0 1.0 Overestimate by 2.5cm 0 35.721 0.0068 9.2% 0.9992 85.91% 100.00%
1.0 1.0 Overestimate by 2.5 cm Underestimate by 1.7 kg 24.557 0.0454 34.4% 0.9995 89.42% 100.00%
1.0 1.0 Overestimate by 5.0 cm Underestimate by 3.2 kg 132.033 0.0000 0.0% 0.9959 73.54% 100.00%
1.0 1.0 Overestimate by 5.0 cm Underestimate by 1.7 kg 168.291 0.0000 0.0% 0.9949 70.73% 100.00%
1.0 1.0 Overestimate by 2.5cm Underestimate by 3.2 kg 272.338 0.0000 0.0% 0.9917 65.31% 100.00%
0.9 0.9 Overestimate by 2.5 cm Underestimate by 1.7 kg 24.011 0.0522 38.0% 0.9989 91.19% 100.00%
0.9 0.9 Overestimate by 5.0 cm Underestimate by 3.2 kg 127.114 0.0000 0.0% 0.9961 74.45% 100.00%
0.9 0.9 Overestimate by 5.0 cm Underestimate by 1.7 kg 161.191 0.0000 0.0% 0.9951 71.82% 100.00%
0.9 0.9 Overestimate by 3.0 cm Underestimate by 3.2 kg 265.706 0.0000 0.0% 0.9924 66.21% 100.00%
0.8 0.8 Overestimate by 2.5cm Underestimate by 1.7 kg 28.423 0.0299 23.6% 0.9958 91.72% 99.88%
0.8 0.8 Overestimate by 2.5cm Underestimate by 3.2 kg 128.995 0.0000 0.0% 0.9938 76.05% 100.00%
0.8 0.8 Overestimate by 2.5 cm Underestimate by 1.7 kg 160.670 0.0000 0.0% 0.9929 73.05% 100.00%
0.8 0.8 Overestimate by 5.0 cm Underestimate by 3.2 kg 267.507 0.0000 0.0% 0.9907 67.18% 100.00%

196
 Doctor of Philosophy Epidemiology (PhD) Dissertation Laura C.A. Rosella

6 g) C-statistics from bias simulations for males and females

C-statistic C-statistic
Height Weight
ICC ICC Bias Height Bias Weight Mean StDev Min Max Mean StDev Min Max
1.0 1.0 0 0 0.6858 0.0082 0.6628 0.7134 0.7179 0.0091 0.6933 0.7424
1.0 1.0 0 Underestimate by 0.5 kg 0.6847 0.0088 0.6623 0.7107 0.7126 0.0090 0.6931 0.7468
1.0 1.0 0 Underestimate by 1.0 kg 0.6834 0.0087 0.6607 0.7098 0.7145 0.0089 0.6918 0.7383
1.0 1.0 0 Underestimate by 1.5 kg 0.6821 0.0086 0.6592 0.7081 0.7127 0.0089 0.6898 0.7369
1.0 1.0 0 Underestimate by 2.0 kg 0.6810 0.0085 0.6556 0.7083 0.7110 0.0088 0.6882 0.7338
1.0 1.0 0 Underestimate by 2.5 kg 0.6798 0.0084 0.6560 0.7072 0.7093 0.0087 0.6880 0.7336
1.0 1.0 Overestimate by 2.5cm Underestimate by 3.0 kg 0.6786 0.0084 0.6531 0.7066 0.7078 0.0085 0.6846 0.7316
1.0 1.0 Overestimate by 2.5 cm 0 0.6856 0.0088 0.6639 0.7118 0.7174 0.0090 0.6940 0.7428
1.0 1.0 Overestimate by 5.0 cm 0 0.6852 0.0087 0.6626 0.7116 0.7169 0.0089 0.6944 0.7422
1.0 1.0 Overestimate by 5.0 cm 0 0.6848 0.0086 0.6616 0.7120 0.7163 0.0088 0.6928 0.7397
1.0 1.0 Overestimate by 2.5cm 0 0.6844 0.0086 0.6590 0.7126 0.7157 0.0089 0.6926 0.7408
1.0 1.0 Overestimate by 2.5 cm 0 0.6841 0.0084 0.6599 0.7111 0.7152 0.0087 0.6929 0.7383
1.0 1.0 Overestimate by 5.0 cm 0 0.6838 0.0083 0.6601 0.7095 0.7147 0.0086 0.6918 0.7391
1.0 1.0 Overestimate by 5.0 cm Underestimate by 1.7 kg 0.6815 0.0085 0.6561 0.7086 0.7118 0.0087 0.6888 0.7353
1.0 1.0 Overestimate by 3.0 cm Underestimate by 3.2 kg 0.6762 0.0079 0.6519 0.7025 0.7037 0.0080 0.6818 0.7263
1.0 1.0 Overestimate by 2.5cm Underestimate by 1.7 kg 0.6776 0.0078 0.6551 0.7039 0.7055 0.0078 0.6845 0.7264
1.0 1.0 Overestimate by 2.5cm Underestimate by 3.2 kg 0.6737 0.0076 0.6520 0.6988 0.7000 0.0075 0.6774 0.7204
0.9 0.9 Overestimate by 2.5 cm Underestimate by 1.7 kg 0.6687 0.0087 0.6428 0.6925 0.7014 0.0089 0.6778 0.7259
0.9 0.9 Overestimate by 5.0 cm Underestimate by 3.2 kg 0.0066 0.0082 0.6379 0.6926 0.6938 0.0081 0.6715 0.7178
0.9 0.9 Overestimate by 2.5cm Underestimate by 1.7 kg 0.0067 0.0079 0.6413 0.6906 0.6958 0.0079 0.6737 0.7173
0.9 0.9 Overestimate by 2.5 cm Underestimate by 3.2 kg 0.0066 0.0000 0.6368 0.6888 0.6903 0.0077 0.6656 0.7104
0.8 0.8 Overestimate by 5.0 cm Underestimate by 1.7 kg 0.6528 0.0090 0.6293 0.6791 0.6894 0.0092 0.6621 0.7163
0.8 0.8 Overestimate by 5.0 cm Underestimate by 3.2 kg 0.6486 0.0084 0.6218 0.6742 0.6821 0.0082 0.6590 0.7036
0.8 0.8 Overestimate by 2.5cm Underestimate by 1.7 kg 0.6500 0.0083 0.6244 0.6796 0.6840 0.0080 0.6622 0.7052
0.8 0.8 Overestimate by 2.5 cm Underestimate by 3.2 kg 0.6467 0.0080 0.6214 0.6771 0.6789 0.0077 0.6547 0.7005

197