EGAPP? Working Group Methods Summary

Archived: This Page Is No Longer Being Updated

This web page is archived for historical purposes and is no longer being maintained or updated.

The independent Evaluation of Genomic Applications in Practice and Prevention (EGAPP™) Working Group reviews the scientific evidence for selected genetic tests and develops recommendation statements about the appropriate use of these tests. The following tables provide summarized information about the methods that the EGAPP™ Working Group uses to conduct this work. These methods have also been published in January 2009 issueExternal of Genetics in Medicine.

Table 1: Test Applications

This table describes categories of genetic test applications and some characteristics of how clinical validity and clinical utility are assessed for each.
Application	Clinical Validity	Clinical Utility
Diagnosis (symptomatic patient)	Association of marker with disorder	Improved clinical outcomes – health outcomes based on diagnosis and subsequent intervention or treatment Availability of information useful for personal or clinical decision-making End of diagnostic odyssey
Disease Screening (asymptomatic patient)	Association of marker with disorder	Improved health outcome based on early intervention for screen positive individuals to identify a disorder for which there is intervention or treatment, or provision of information useful for personal or clinical decision making
Risk assessment/susceptibility	Association of marker with future disorder (consider possible effect of penetrance)	Improved health outcomes based on prevention or early detection strategies
Prognostic	Association of marker with natural history benchmarks of the disorder	Improved health outcomes, or outcomes of value to patients, based on changes in patient management
Pharmacogenomic Predicting treatment response or adverse events	Association of marker with a phenotype/metabolic state that relates to drug efficacy or adverse drug reactions	Improved health outcomes or adherence based on drug selection or dosage

Top of Page

Table 2: Components of Evidence Evaluation

Because of the newness of the field of genetic testing, direct evidence to answer an overarching question about the effectiveness and value of testing is rarely available. Therefore, the EGAPP™ Working Group evaluated evidence in three key areas to construct a chain of evidence to address the overarching question.
Analytic Validity (Technical Performance)	Clinical Validity (Strength of Clinical Correlation)	Clinical Utility (Impact on Health Outcomes)
Tests ability to accurately and reliably measure analyte or genotype of interest. Sensitivity Specificity Assay robustness Quality control	Test’s ability to accurately and reliably identify or predict the disorder of interest. Sensitivity Specificity Positive Predictive Value Negative Predictive Value	Likelihood that using the test to guide management will significantly improve health-related outcomes. Benefits vs. Harms Added value compared to not using test Effectiveness Efficacy

Top of Page

Table 3: Quality of Evidence

This table describes how the quality of evidence was graded in terms of its adequacy to address the key questions of each of the evidence components: analytic validity, clinical validity, and clinical utility.
Quality of Evidence
Adequacy of Information to Answer Key Questions	Analytic Validity	Clinical Validity	Clinical Utility
Convincing	Studies that provide confident estimates of analytic Sensitivity and specificity using intendedsample types from representative populations Two or more Level 1 or 2 studies that are generalizable, have a sufficient number and distribution of challenges, and report consistent results One Level 1 or 2 study that is generalizable and has an appropriate number and distribution of challenges	Well-designed and conducted studies in representative population(s) that measure the strength of association between a genotype or biomarkerand a specific and well-defined disease or phenotype Systematic review/meta-analysis of Level 1 studies with homogeneity Validated Clinical Decision Rule High quality Level 1 cohort study	Well-designed and conducted studies in representative population(s)that assess specified health outcomes Systematic review/meta-analysis of randomized controlled trials showing consistency in results At least one large randomized controlled trial (Level 2)
Adequate	Two or more Level 1 or 2 studies that Lack the appropriate number and/or distribution of challenges Are consistent, but not generalizable. Modeling showing that lower quality (Level 3, 4) studies may be acceptable for a specific well-defined clinical scenario	Systematic review of lower quality studies Review of Level 1 or 2 studies with heterogeneity Case-control study with good reference standards Unvalidated Clinical Decision Rule (Level 2)	Systematic review with heterogeneity One or more controlled trials with out randomization (Level 3) Systematic review of Level 3 cohort studies with consistent results
Inadequate	Combinations of higher quality studies that show important unexplained inconsistencies One or more lower quality studies (Level 3 or 4) Expert opinion	Single case-control study Nonconsecutive cases Lacks consisitently applied reference standards Single Level 2 or 3 cohort/case-control study Reference standard defined by the test or not used systematically Study not blinded Level 4 data	Systematic review of Level 3 quality studies or studies with heterogeneity Single Level 3 cohort or case-control study Level 4 data

Note: The “levels” used in ranking studies are described in Table 4.
Top of Page

Table 4: Ranking of Data Sources/Study Designs for Components of Evaluation

This table illustrates how data sources and study designs were ranked in order to assess the quality of evidence for each component of the evidence evaluation.
Level^a	Analytic Validity	Clinical Validity	Clinical Utility
1	Collaborative study using a large panel of well characterized samples Summary data from well-designed external proficiency testing schemes or interlaboratory comparison programs	Well designed longitudinal cohort studies Validated clinical decision rule^b	Meta-analysis of randomized controlled trials (RCT)
2	Other data from proficiency testing chemes Well designed peer-reviewed studies (e.g., method comparisons, validation studies) Expert panel reviewed FDA summaries	Well designed case-control studies	A single randomized controlled trial
3	Less well designed peer-reviewed studies	Lower quality case-control and cross-sectional studies Unvalidated clinical decision rule^b	Controlled trial without randomization Cohort or case-control study
4	Unpublished and/or non-peer reviewed research, clinical laboratory, or manufacturer data Studies on performance of the same basic methodology, but used to test for a different target	Case Series Unpublished and/or non-peer reviewed research, clinical laboratory or manufacturer data Consensus guidelines Expert opinion	Case Series Unpublished and/or non-peer reviewed studies Clinical laboratory or manufacturer data Consensus guidelines Expert opinion

^aHighest level is 1.

^bA clinical decision rule is an algorithm leading to result categorization. It can also be defined as a clinical tool that quantifies the contributions made by different variables (e.g., test result, family history) in order to determine classification/interpretation of a test result (e.g., for diagnosis, prognosis, therapeutic response) in situations requiring complex decision-making.

Top of Page

Table 5: Recommendation Classification

This table demonstrates the possible recommendations derived from the evaluation of the evidence components, the overall level of certainty of net health benefits, and contextual factors.
Level of Certainty	Recommendation
High or Moderate	Recommend for . . . . . . if the magnitude of net benefit is substantial, moderate, or small, unless additional considerations warrant caution.Recommend against . . . . . . if the magnitude of net benefit is zero or there are net harms.
Low	Insufficient evidence . . . . . . if the evidence for clinical utility or clinical validity is insufficient in quantity or quality to support conclusions or make a recommendation.

Disclaimer:
Our office does not offer medical advice to individuals. If you have specific concerns about your health, please discuss them with your doctor.

Top of Page

Page last reviewed: October 21, 2011 (archived document)

Content source: National Center on Birth Defects and Developmental Disabilities, Public Health Genomics Branch in the Division of Blood Disorders and Public Health Genomics