15 views

Uploaded by Susi Rutmalem

Meta

- Study Questions Epidemiology
- Home Task_ Systematic Review Evaluation
- Study Links Teen's Cell Phone Use With Greater Diabetes, Obesity Risk
- CD005978 Standard
- Fumo e implantes
- Pneumonia
- Valentine Et Al. 2010 How Many Studies Do You Need - A Primer on Statistical Power for Meta-Analysis
- Acupuncture for OA Knee
- Research in Developmental Disabilities Volume 32 Issue 6 2011 [Doi 10.1016%2Fj.ridd.2011.05.014] Kari-Anne B. Næss; Solveig-Alma Halaas Lyster; Charles Hulme; M -- Language and Verbal Short-term Memor
- Energy Requirements of Adult Cats -Meta-Analysis
- Video game and physical aggression
- 필라테스와 멘탈트레이닝
- National Center for Family Literacy Report 2009
- 10.1007%2Fs11159-011-9198-8.pdf
- Methotrexate and Lung Disease in Rheumatoid Arthritis
- cochrane amalgama.pdf
- Mineral Requirements of Fish a Systematic Review
- Neuroscience Dyslexia[1]
- Peerj Reviewing 13867 v0
- Homeopathy

You are on page 1of 98

Partini Pudjiastuti

Child Health Department

Faculty of Medicine University of Indonesia

Background

Facts: INFORMATION EXPLOTION

as new information produced every year

isn't very good (= poor quality)

is derived from different methods & definitions (= poor

standardization)

is often contradictory (= heterogeneity)

information?

all areas that might be relevant to their practices.

New methods to synthesize and present

information from widely dispersed publications

are needed

. . . .

Jerome Kassirer. Clinical trials and meta-analysis: what do they

do for us? N Engl J Med 1992; 327:273-4.

How do we summarize

medical information?

Traditional Approach

Expert Opinion

Narrative review articles (Literature review)

Consensus statements (group expert opinion)

Explicit quantitative synthesis of ALL the evidence

Feature

Narrative Review

Systematic Review

search strategy databases that were used may

not be provided

Authorship A recognized expert(s) on the

topic

Article selection Typically not specified

criteria

and clinical expertise

Consistently applied inclusion and

exclusion criteria

locate literature on the topic

area in question

Appraisal of Indefinite, may be variable

included articles

Synthesis A qualitative summary is

usually provided

primary studies on a particular research

question

Critical appraisal is meticulous,

typically involving the use of

data extraction forms

A qualitative summary is provided,

quantitative when the data can be

pooled

Evidence-based Chiropractic

comprehensive with a list of all

databases that were utilized

Usually evidence-based

literature review designs

Narrative Review

a specific topic.

Does not follow strict systematic methods to locate

and synthesize articles.

Systematic Review

that the maximum extent of relevant research has

been considered.

Original articles are methodologically appraised and

synthesized.

Meta-analysis

are the result of a systematic literature review.

Capable of performing a statistical analysis of the

pooled results of relevant studies.

Meta-Analysis

Definition:

A specific subset of systematic reviews that

order to find a common effect.

analysis results from individual studies for the

purpose of integrating findings (Glass, 1976).

Meta-Analysis

Definition:

Meta-analysis is a name that is given to any

independent studies are combined statistically

to produce a single estimate of the effect of

a particular intervention or health care

situation (Jadad, 1998)

Considered as a true investigation:

observational - retrospective study

Review

Systematic

review

Meta-analysis

Meta-Analysis

Initially developed in social sciences in mid-1960s

Adapted to medical studies in early 1980s

Initially applied to RCTs esp. when individual

Also applied to observational epidemiologic studies

often with little fore-thought which generated

much controversy

Explosion in the number of published metaanalyses in the last 10-15 years.

10

1. to summarize a large and complex body of literature on

2.

3.

4.

5.

a topic

to resolve conflicting research reports in the literature

to clarify or quantify the strengths and weaknesses of

studies on a topic

to document the need for a major clinical trial

to avoid the time and expense of conducting a clinical

trial

6. to increase statistical power by combining many smaller studies:

Statistical power refers to the probability of detecting a finding (e.g

combining studies provides a larger sample increase in statistical

power

Increase the evidence for, or confidence in, a finding/conclusion

Precision refers to the degree of accuracy in the estimation, usually

estimate (i.e. Precision =1 / standard deviation)

8. to detect smaller treatment effects that have been

reported

9. to investigate variations in treatment effects through

subgroup (or stratified) analysis

10. to investigate or improve the generalizability (external

validity) of known treatment effects

Hierarchy of studies

analysis

Require careful planning, and adequate

Need to develop study protocol

INTRODUCTION

Background

/ Justification

Research question(s)

Objectives

Hypothesis

METHODS

Steps in meta-analysis

1. Identification:

Formulate question

Develop search strategy

Searching relevant studies

2. Selection:

Choosing variables (comparison & outcomes)

3. Abstraction:

4. Analysis:

Quantitative pooling

Sensitivity analysis

Investigate heterogeneity & biases

5. Write up:

Clear presentation

1. IDENTIFICATION

The research question must be clearly defined:

Defined patient population

Clear & consistent definitions of the disease,

studies

IDENTIFICATION - Sources

M-As use systematic, explicit search procedures (cf.

MEDLINE

4100 journals

1966 - present

other search engines: BRS Colleague, WinSPIRs, etc

EMBASE

Expensive, not widely available in US

Identification - sources

Cochrane Collaboration Controlled Trials Register

Over 160,000 trials, including abstracts (+ translations)

by subscriptions.. MSU Electronic Library database

includes

MEDLINE,

EMBASE

non-English publications

non-indexed publications

hand-search of journals

Other MEDLARS

Index Medicus

important if searching before 1966

hand-search only

Identification - steps

Search own personal files

Search electronic databases

Review titles and on-line abstracts to eliminate irrelevant

Retrieve remaining articles, review, and determine if meet

inclusion/exclusion criteria

Consult experts/colleagues/companies

Conduct hand-searches of non-electronic databases and/or relevant journals

Consider consulting an expert (medical librarian) with training in MEDLINE

Identification

find relevant study

Use appropriate synonyms

Use electronic databases, hand searching, etc

LOOK FOR UNPUBLISHED PRIMARY STUDIES

completed/ongoing trials)

Non-English Studies

MEDLINE

Translation

Quality

Excluding non-English studies can effect conclusions

But including means you need a translation just to determine

eligibility!

Fugitive Literature

unpublished

dissertations

drug

company studies

book

chapters

non-indexed

conference

proceedings

government

reports

pre-MEDLINE

(1966)

2. SELECTION

TYPICAL INCLUSION CRITERIA:

study design (e.g., RCTs?, DBPC?, Cohort & CCS?)

setting (emergency department, outpatient, inpatient)

age (adults only, > 60 only, etc)

year of publication or conduct (esp. if technology or typical dosing

changes)

similarity of exposure or treatment (e.g., drug class, or dosage)

similarity of outcomes (case definitions)

minimum sample size or follow-up

languages?

complete vs incomplete (abstracts)

published vs fugitive?

pre-1966?

Selection Other

Issues.

Multiple publications from same study?

Include only one!

Selection process should be done independently by

at least 2 reviewers

Measure agreement (K) and resolve

discrepancies

for exclusion

Cannot

others

include all studies

Keep the ones with

high levels of evidence

good quality

check with QUOROM guidelines

Case series, and case reports definitely out

Selection problems are major problems

Detailed

Guidelines

A Good Checklist

Use it for reporting

Meta Analysis

Systematic reviews

and selection

Non-steroidal anti-inflammatory drugs,

ncluding cyclo-oxygenase-2 inhibitors,

in osteoarthritic knee pain:

meta-analysis of randomised

placebo controlled trials

3. ABSTRACTION

Goal: to abstract reliable, valid and bias free information from all

written sources

Should expect a degree of unreliability

intra- and inter- rater reliability is rarely if ever 100%!!

Article may be wrong due to typographical or copyediting errors

Reported results can be misinterpreted

Errors in data entry during abstraction process

Weighting

Size (sample size (n))

Event rate

Homogeneity (inverse of the variance)

Quality

Other factors

Abstraction

Typical process

2 independent reviewers

Practice with 2 or 3 articles to

calibrate

Use a 3rd reviewer or consensus

meeting to resolve conflicts

Measure agreement (K) and resolve

discrepancies

Abstraction investigator

bias

Abstractor may be biased in favor of (or against!)

finding), or researcher/institution, or journal.

review)

Remove study information that could affect inclusion or

author, title, journal, institution, country

Berlin (97):

compared blinded vs non-blinded reviews

Found discrepancy in which studies to include but

Time consuming

Probably can avoid esp. if use well defined abstraction

procedures

Abstract Data:

Create a spreadsheet (Excel)

For each study, create the following columns:

name of the study

name of the author, year published

number of participants who received intervention

number of participants who were in control arm

number who developed outcomes in intervention

number who developed outcomes in control arm

Spreadsheet Data

We created seven

columns:

trial: trial identity code

trialname: name of trial

year: year of the study

pop1: study population

deaths1: deaths in study

pop0: control population

deaths0: deaths in control

4. ANALYSIS

Many techniques

Still controversial

Ask expert (a statistician) in M-A

Focus on variability (heterogeneity)

Analysis - heterogeneity

What is Heterogeneity?

No two studies are exactly the same! Heterogeneity (or Diversity) is

the extent to which individual studies vary from one another.

Types of Heterogeneity

1. Clinical (for example, studies with different age

criteria for enrollment)

2. Methodological (variability in design for example,

parallel group vs. crossover placebo controlled trial)

3. Statistical (observed differences in treatment effects among

studies are greater than those one would expect by chance alone)

Analysis - heterogeneity

Causes of heterogeneity

Differences between studies with respect to:

Patients: diagnosis, prognostic factors, in- and exclusion

criteria, etc.

Interventions: type, dose, duration, etc.

Outcomes: type, scale, cut-off points, duration of follow-up,

etc.

Quality and methodology: randomised or not, allocation

Heterogeneity

recommended)

Test for it and do not pool results if studies

are significantly heterogeneous

Incorporate it:

assume that heterogeneity is due to

random differences among studies whose

sources can not be identified

use random effects model

Consequences of

heterogeneity

When the results of studies in a meta-analysis are

The meta-analysis may actually be worthless if too

dissimilar

For instance, combining studies that used different types

of comparison groups

Or outcomes that were dissimilar

ANALYSIS

Focus on variability (heterogeneity)

Fixed effects model

o only intra-study, ignore interstudy var

o narrower CI

Random effects model

o considers also inter-study var

o broader CI

Fixed Effects Model

studies

Pooling: Mantel Haenszel OR

Test: test of heterogeneity

If significant, go for random effects

model

Short 95% CI for summary

Smaller summary estimate

OR=0.77 [0.72,0.83]

Assume that TRUE log odds ratio

comes from a normal distribution

Method: DerSimonian Lairs

method (DSL) of calculating Odds

Ratio

OR=0.78 [0.69,0.88]

Meta-analysis (Forest)

plot

1. Eyeball test do they look the same?

2. Test of Null hypothesis of no variation (p-value)

3. Proportion of variation not due to chance (I2)

The Chi2 test measures the amount of variation

in a set of trials, and tells us if it is more than

would be expected by chance

Small p values suggest that heterogeneity is

present

This test is not very good at detecting

heterogeneity. Often a cut-off of p<0.10 is used,

but lack of statistical significance does not mean

there is no heterogeneity

44

I2 is the proportion of variation that is due to

heterogeneity rather than chance

Large values of I2 suggest heterogeneity

Roughly, I2 values of 25%, 50%, and 75% could

be interpreted as indicating low, moderate, and

high heterogeneity

(Higgins JPT et al. Measuring inconsistency in meta-analyses. BMJ .

2003;327:557-60.)

45

Test of sensitivity:

Was the result robust?

Random & fixed effects: similar

Exclusion of low weighted studies: similar

? Publication bias: larger sample size gave smaller

effect size

Exclusion of special cases, e.g. incomplete follow-up

Subgroup analysis

Meta-analyses typically include patients

e.g., age, gender, condition severity,

patient history, etc.

Patients in these subgroups may respond

to treatment differently

e.g., low back pain patients with leg pain

may respond to treatment differently

than low back pain only patients

Subgroup analysis

(cont.)

or findings

The process helps readers to distinguish

the effects of a treatment between

subgroups

The statistical power of the subgroups will

decline as a result

Analyse Data

Statistically

Effect Size (Odds Ratio)

Variance with 95% Confidence Interval

Test of Heterogeneity

Two Graphs

Forest Plot

Funnel Plot

are

Use Statistical Packages, several choices

Summary Estimates

Mantel Haenszel OR=0.77

95% Confidence Interval

[0.72, 0.83]

Test of Heterogeneity:

Chi-square (df=21) = 31.5

P Value = 0.07

that those receiving

streptokinase at AMI are about

77% at risk of death (23% less

likely to die)

That in 95 out of 100 such

meta analyses, the pooled

Odds Ratio would lie between

0.72 and 0.83, indicating a

statistically significant

protective effect

That these studies were not

significantly heterogeneous

Forrest Plot

For numeric outcome

Presented as mean difference for individual

Represented by square, the area = relative

weighted value

Summarized as combined mean difference with

CI as diamond

Vertical line: mean diff = 0 (no effect)

Vertical dotted line: if CI crosses this line:

means homogenous; otherwise: heterogeneous

of RCTs with numerical outcome (xe-xc)

Study I

Study II

Study III

Study IV

Study V

Study VI

Study VII1

Study VIII

1992

1994

1995

1995

1996

1997

1999

2000

Combined

-0.3

Favor drug

Xe-xc=0

+0.3

Favor placebo

Study results

For nominal outcomes:

Presented as OR for individual study with

corresponding 95% CI

weighted value

Vertical line: OR = 1 (no effect)

Vertical dotted line: if CI crosses this line: means

of RCTs with nominal outcome

Study I

Study II

Study III

Study IV

Study V

Study VI

Study VII1

Study VIII

1992

1994

1995

1995

1996

1997

1999

2000

Combined

0.1

10

OR = 1

Favor drug

Favor placebo

Forest plot

The label tells you what the comparison and

Forest plot

Each study has an ID

Forest plot

The data for each trial are here, divided into

Forest plot

Results from each trial are also given

numerically

Forest plot

This is the % weight given to each study in

Forest plot

The size of the block for each study is

line represents the confidence interval.

Forest plot

The label above the graph tells you which

Forest plot

Here, towards the left, the scale is less

than one, meaning risperidone has

made people less likely to leave the

study early

Forest plot

The vertical line in the middle is where

effect there is no difference between

the two

Forest plot

treatment is better than the control

Forest plot

The pooled analysis is given a diamond

point estimate and the horizontal width

in the confidence interval. Does the

confidence interval cross the line of no

effect?

Forest plot

Results of the pooled analysis are also given

numerically

Forest plot

The result of a Z test with a p value are also

Forest plot

The forest plot displays the results of the Chi-

Publication bias

Published studies are not representative of all studies

that have been performed

Articles with positive findings (P < 0.05) are more

likely to be published

Hence published studies are a biased sub-set

Publication bias = systematic error of M-A that results

from using only published studies

Easterbrook (1991): 285 analyzed studies reviewed by Oxford

Ethics Committee 1984-87

Study Status

% (P < 0.05)

Published

138

67%

Presented Only

69

55%

Neither

78

29%

Total

285

54%

Simes (1986): Chemotherapy for Advanced Ovarian CA

Comparison of Published Trials vs Registered Trials

Results

Published

Registered

16

13

1.16

1.06

95% CI

1.06 1.27

0.97 1.15

P value

0.02

0.24

N

Median Survival

Ratio

Funnel Plots

Funnel plots are a device for checking for publication bias.

Each dot represents the overall effect

from one RCT.

As sample size increases, the width of

the confidence interval should

decrease.

Result should be located in a

symmetric, triangular area centered on

the overall effect for all studies.

Funnel Plots

Missing studies will manifest as an asymmetry in the

funnel plot.

Missing studies will appear

as a gap in the portion of the

funnel plot where you would

expect to find negative

studies.

The unopposed positive

studies will shift the

apparent treatment effect

(blue line) towards a larger

effect size than it really is.

Cumulative Meta-analysis

75

harmful?

A topic of great debate and controversy for many years

First few epidemiologic studies were published in 1918

Vigorously attacked by the tobacco industry

Too small an association

Potential bias

Potential confounding

Lack of biological proof

It was not until about 10 years ago when several

official bodies/agencies concluded that passive

smoking is a cause of lung cancer

The tobacco industry continues to dispute this claim!!

Hackshaw AK. Stat Meth Med Res 1998;7:119-136.

Hackshaw et al. conducted a very

comprehensive systematic review in 1997:

They identified 37 published studies that reported

risk of lung cancer among lifelong non-smoking

women according to the husbands smoking

status

Their meta-analysis revealed that the overall risk

of lung cancer among lifelong non-smoking

women was 1.24 times higher when their

husbands smoked, as compared to those women

whose husbands did not smoke.

Hackshaw AK et al. BMJ 1997;315:980-88.

Hackshaw AK. Stat Meth Med Res 1998;7:119-136.

Passive

smoking and

lung cancer

review

Hackshaw AK. Stat Meth Med Res 1998;7:119-136.

Hackshaw AK. Stat Meth Med Res 1998;7:119-136.

Total 37 studies

Is passive

smoking

harmful?

Sub-group analysis

Hackshaw AK. Stat Meth Med Res 1998;7:119-136.

Criticism of Meta-analysis

The file drawer problem/Publication bias: There is always a high

likelihood for studies that did not achieve statistical significance not

to be published. This is called the file drawer problem because it is

assumed that this studies are filed away somewhere and not

accessible to the public. Therefore statistically significant studies

are more likely to be included in the meta-analysis which will result

in an overestimate of the treatment effect.

Potential solutions:

if possible contact all persons known to work in the field to

inquire about studies done, but not published

estimate how many studies it will take to reduce the treatment

effect to non-significance

Criticism of Meta-analysis

The Simpsons Paradox: A reversal in the direction of the

sources are combined

Confounding variables: There is always potential for

differences across studies that may be confounded with

treatments used. For example, if studies in a meta-analysis

were done in different countries, cultural differences may be

confounded with treatment differences.

Subtle differences in treatment with same name

Criticism of Meta-analysis

Biased or flawed studies: If the studies in the meta-analysis are

Statistical significance versus practical / clinical significance: Metaanalysis is likely to find statistical significance because of the

increase in power as the sample size increases. Must evaluate the

magnitude of the treatment effect to assess clinical significance

False findings of no difference: A statistically non-significant result

may also be found because one does not have enough data (or

power). But one must remember that no evidence of effect is NOT

evidence of no effect

Rule of Thumb: Always determine the sample size whenever no

Evaluating metaanalysis

Validity

Importance

Applicability

COMMENTS

Did the meta-analysis address a

focused question?

the title or abstract?

Specify the subjects, exposures,

articles for inclusion appropriate? and outcomes? Specify the

methodological standards used

to select studies?

Is it likely that important relevant

studies were missed?

search? Identify unpublished

studies?

GUIDE

COMMENTS

studies appraised?

Were the assessments of

studies reproducible?

participate to select studies?

study to study?

Tests of Homogeneity?

the meta-analysis?

their size? RR and the absolute

risk reduction? Likelihood

Ratios? ORs? Effect Size?

GUIDE

COMMENTS

How precise were the results?

patient care?

Were all clinically important

outcomes considered?

Are the benefits worth the harm

and costs?

estimate?

Thank you

Funnel Plots

Treatment effects (for eg. lnOR, lnRR, RD) is plotted

against sample size (or other measures of precision such

as standard error or variance)

5000

10000

Sample size

4000

3000

2000

1000

0

0.01

1000

100

10

0.1

10

0.01

0.1

10

Funnel Plot

Plots the effect size against

the sample size of the study

To study a funnel plot, look at

its LOWER LEFT corner, thats

where negative or null studies

are located

If EMPTY, this indicates

PUBLICATION BIAS

Plots

Scatterplot of effect estimates against sample size

Used to detect publication bias

If no bias, expect symmetric, inverted funnel

x

x

x

x x x

x

x

x x x

x

x

x

x x x

x x x x

Missing studies = small effects size with

negative findings

X

X

Sample

Size

(precision)

X

X

X

X

X

X

X

X

X

X

Effect Size

X

X

X

X

1. Attempt to Retrieve all Studies

Required for Cochrane Publications

Difficult to identify unpublished studies and then to find out details about

them

Number of unpublished negative studies to negate a positive metaanalysis:

X = [N x (ES) / 1.645]2 - N

where: N = number of studies in meta-analysis,

ES = effect size

Example:

If N = 25, and ES = 0.6 then X = 58.2

Almost 60 unpublished negative studies would be required to negate the

meta-analysis of 25 studies.

Growth of Meta-analysis

3000

Number of Publications

2500

2000

1500

1000

500

0

93-94

94-95

95-96

96-97

97-98

98-99

99-00

2000-1

2001-2

2002-3

2003-4

Year of Publications

meta-analysis as publication type from PubMed, from years 1993 through 2004

Fugitive Literature

Sometimes important sources of information

Hard to track down contact experts/colleagues

Need to decide whether to include or not - general

consensus is that you should.

Example

Let's say we want to know whether

streptokinase is protective for death

from acute myocardial infarction. How

should we set up a search strategy?

The Search

streptokinase[text word] OR acute

myocardial infarction[text word] produces ALL

articles that contain EITHER streptokinase OR

acute myocardial infarction anywhere in the

text inclusive, many

streptokinase [text word] AND acute

myocardial infarction [text word] will capture

only those subsets that have BOTH

streptokinase AND acute myocardial

infarction anywhere in the text restrictive,

few

Summary Estimates

Mantel Haenszel OR=0.77

95% Confidence Interval

[0.72, 0.83]

Test of Heterogeneity:

Chi-square (df=21) = 31.5

P Value = 0.07

that those receiving

streptokinase at AMI are about

77% at risk of death (23% less

likely to die)

That in 95 out of 100 such

meta analyses, the pooled

Odds Ratio would lie between

0.72 and 0.83, indicating a

statistically significant

protective effect

That these studies were not

significantly heterogeneous

- Study Questions EpidemiologyUploaded bythechickendidwhat
- Home Task_ Systematic Review EvaluationUploaded byOmar Ayash
- Study Links Teen's Cell Phone Use With Greater Diabetes, Obesity RiskUploaded byCBS Local
- CD005978 StandardUploaded byMuhammad Hadi Arwani
- Fumo e implantesUploaded byLuis Fernando Rodrigues
- PneumoniaUploaded byFerry Ghifari
- Valentine Et Al. 2010 How Many Studies Do You Need - A Primer on Statistical Power for Meta-AnalysisUploaded byFabianto Santoso
- Acupuncture for OA KneeUploaded bydtncorreo
- Research in Developmental Disabilities Volume 32 Issue 6 2011 [Doi 10.1016%2Fj.ridd.2011.05.014] Kari-Anne B. Næss; Solveig-Alma Halaas Lyster; Charles Hulme; M -- Language and Verbal Short-term MemorUploaded bymajose
- Energy Requirements of Adult Cats -Meta-AnalysisUploaded bygabrielwerneck
- Video game and physical aggressionUploaded byMichelleMartina
- 필라테스와 멘탈트레이닝Uploaded bykang soon chul
- National Center for Family Literacy Report 2009Uploaded byvmeeder
- 10.1007%2Fs11159-011-9198-8.pdfUploaded bymsa11pk
- Methotrexate and Lung Disease in Rheumatoid ArthritisUploaded byamere
- cochrane amalgama.pdfUploaded byFabianni Apolonio
- Mineral Requirements of Fish a Systematic ReviewUploaded byCristian Illesca
- Neuroscience Dyslexia[1]Uploaded byBenjamín Vega Peñate
- Peerj Reviewing 13867 v0Uploaded byDrima Edi
- HomeopathyUploaded bycinderella1977
- rev_TBA1Uploaded byslermalara
- Christakis 2000Uploaded byFajar Yuniftiadi
- HemorrhoidUploaded byMorna gresella
- The Effects of BehavioralCognitive-Behavioral Programs on RecidivismUploaded bybontes67
- Bone Replacement GraftsUploaded byPragati
- Reconsidering Community Based Health Promotion.pdfUploaded bypabloignacio
- 22042_ftpUploaded byAchmad Deza Farista
- A systematic review and meta-analysis of the reliability and validity of sensorimotor.pdfUploaded byDiego D Toledo
- Types, Architecture and Hierarchy of Research StudiesUploaded byEzzeldin Sakr
- Cesarean Delivery and Cerebral Palsy a SystematicUploaded byAlejandra Pinillos Rojas

- The Teachers Guide to Common Learning ChallengesUploaded bySusi Rutmalem
- George-West-AAP-validation-AHD-2001.pdfUploaded bySusi Rutmalem
- NeglectedUploaded bySusi Rutmalem
- A.1-ETHICS-072012Uploaded byOicirtap Arbal Zepol
- A.1-ETHICS-072012Uploaded byOicirtap Arbal Zepol
- A.1-ETHICS-072012Uploaded byOicirtap Arbal Zepol
- Pollak 2013Uploaded bySusi Rutmalem
- dyscalculia-160621081110Uploaded bySusi Rutmalem
- Baca Ini Kamu Pasti Dapat Beasiswa Kuliah Ke Luar NegeriUploaded byDago Dago
- A.1-ETHICS-072012Uploaded bySusi Rutmalem
- A.3-CLINICAL-MODELS-CLASSIFICATION-072012.pdfUploaded bySusi Rutmalem
- Attachment Theory[1]Uploaded bySusi Rutmalem
- A.1-ETHICS-072012Uploaded bySusi Rutmalem
- PENELITIAN TW.docxUploaded bySusi Rutmalem
- Revision NotesUploaded bySusi Rutmalem
- APA_DSM-5-Intellectual-Disability (3).pdfUploaded bykhanzaisdiharana
- winters2007 (1).pdfUploaded bySusi Rutmalem
- 0502009001Uploaded bySusi Rutmalem
- 10.1007_s11920-017-0754-1Uploaded bySusi Rutmalem
- 3. Barragan, 2012Uploaded bySusi Rutmalem
- 0502016082Uploaded bySusi Rutmalem
- 0502016082Uploaded bySusi Rutmalem
- ppfethicsanddx-130519201122-phpapp02Uploaded bySusi Rutmalem
- 3. Ozonoff, 2015Uploaded bySusi Rutmalem
- 3. Takazawa, 2015Uploaded bySusi Rutmalem
- 1. Salloum, 2014Uploaded bySusi Rutmalem
- 0502008009Uploaded bySusi Rutmalem
- 5. Patel, 2014Uploaded bySusi Rutmalem
- New Terminology for Mental Retardation in DSMUploaded bySusi Rutmalem
- e014158.fullUploaded bySusi Rutmalem

- demographic.pdfUploaded byRoxii Mihaela Ivan
- BMW(Chapter7)Uploaded byraghav4231
- Civil Engineering colleges direct admission in BangaloreUploaded byKiran Consultancy
- ProQuestDocuments 2017-03-16Uploaded byAnonymous AIXYBfYB
- Global Optimization With MATLAB Products(Draft)_MathWorksUploaded bycansu
- Silicon India Jan 11 IssueUploaded byPrad Kd
- BingUploaded bydaniel 222016002
- Mathematical Modelling with Sinusoidal Equations DANIEL LENG.docxUploaded byDanny Leng
- Annotated Bibiliography ResearchUploaded bypem
- Declaration SSCUploaded byMaple Meesuwan
- How is 21st century spirituality constructed and reconstructed by performanceUploaded byS Goldingay
- Script for SchneiderUploaded byCj Mizona
- Jadad ScoreUploaded byLira Mirandus Manik
- National Science Foundation: ar 1970 sec3Uploaded byNSF
- making good choices 17 - 18Uploaded byapi-233188563
- TriUploaded byLeandro Andraos
- varliklar05Uploaded byr11111234
- Ptransit Design Doc v0.6Uploaded byChera Robinson
- UNPSReportUploaded byHiina Ch
- Efficacy of Non-speech Oral Motor Exercises for Developmental Spe (1)Uploaded byCristián Leal
- Energy Crisis and Household’s Perception about Solar Energy Acceptance: District Hyderabad, PakistanUploaded byFarrukh Baig
- Detailed Syllabus of Personality Development CourseUploaded byImprint Training Center for Personality development
- (Decision Engineering 1) Lixing Yeung, CKM Lee (Auth.), Hing Kai Chan, Fiona Lettice, Olatunde Amoo Durowoju (Eds.)-Decision-Making for SuUploaded byMarsha Aulia Hakim
- Review on Epidemic of ObesityUploaded bySebastián Fischer Broussain
- Cinetica Reaccion FeCl3 Con KIUploaded byAngie Riobamba
- Report Writing - JNTUUploaded bypavan2091
- Test Internal EnvironmentUploaded bynahanuna
- BBA SyllabusUploaded byVaibhav Misra
- Equilibrio L-V a alta presionUploaded byDaniel Solar
- Scientific MalaysianUploaded byTim Wong