By Woojung Lee and Boshen Jiao
Why do we want to know how real-world evidence (RWE) is being used in health technology assessment (HTA) and what is RWE anyway?
Based on the FDA’s definition, RWE is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from the analysis of real-world data. Real-world data is defined as data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.
There has been a growing interest in incorporating RWE into the HTA process to assess the clinical outcomes and economic value of drugs, mainly due to the limited availability of evidence from randomized clinical trials (RCTs) and the desirable property of RWE in reflecting outcomes in the real-world setting. Despite RCTs being the gold standard, long-term drug effects and economic outcomes are often challenging to collect from the RCTs. The US FDA’s recently released guidance in 2018 on how drug companies may communicate health care economic information to payers and formulary committees that enable the use of off-label evidence related to approved indications also likely increased the use of RWE in economic analysis. Furthermore, US payers increasingly recognize the value of RWE in informing healthcare decision making, believing that RWE, while not a replacement for RCTs, can provide the best available evidence for a HTA in situations when well-controlled clinical trials are difficult to implement.
However, it remains unclear how RWE can be incorporated into the HTA of pharmaceuticals in the US (there has been a few studies in Europe though!). We thought that a quantitative examination of RWE use in the US may give US payers an objective view of the possible roles that RWE can play in the HTA of drugs. This is especially important considering that the use of RWE varies from payer to payer and is generally limited because of the lack of understanding of how to incorporate RWE into the value assessment process. A better understanding of the current roles of RWE can help facilitate its use in the HTA.
What did we do?
We conducted two studies. In the first study, we assessed the use of RWE in the economic assessment of drugs by the Institute for Clinical and Economic Review (ICER), a nonprofit independent research organization that evaluates the clinical and economic values of medical interventions. Their reports have been increasingly used by the US payers in their decision-making processes. Specifically, we reviewed the long-term cost-effectiveness analysis (CEA) and potential budget impact analysis (BIA) sections of final evidence reports published by the ICER between January 2014 and June 2019 and extracted evidence of RWE use.
We identified 407 RWE uses in CEA and BIA sections of the ICER reports, which accounted for 33% of all model inputs (i.e. 67% of model inputs were not informed by RWE). However, this proportion ranged from 4% – 77% for each final evidence report, showing a large variance. (Figure)
We found that over time (from 2014-2019) there is an increase in the use of RWE to inform model inputs. The most common reasons were to inform mortality or disease progression rate (29%) and health care costs (21%) (Figure).
Drug-specific clinical inputs such as drug effectiveness, drug-specific discontinuation rate, and adverse drug events were rarely informed by RWE (<3%). The most frequently used study design was a retrospective cohort (50%) followed by a prospective cohort study (17%). In terms of the data sources, registry data were the most frequently used data (40%), followed by administrative claims data (18%) and patient survey/diary data (18%).
Nondrug-specific clinical inputs (e.g., disease progression and mortality rate, patient characteristics, and incidence) and economic inputs (e.g., health care costs and treatment pattern) were mostly informed by retrospective studies whereas drug-specific clinical inputs (e.g., drug effectiveness and adverse drug events) were more likely to be informed by prospective studies. In terms of the data sources, registry data and administrative claims data were the main sources of nondrug-specific clinical inputs and economic inputs. Drug-specific clinical inputs were informed by diverse sources including electronic health records, claims, as well as registry data. Not surprisingly, only 1% of RWE were pre-registered (one meta-analysis and 4 prospective studies). We also found that about 30% of RWE was sponsored by industry.
In the second study, we assessed the use of RWE in ICER’s scoping and comparative clinical effectiveness (CCE) assessments. We examined the frequency of use, trends, and reasons of RWE use overall and further stratified by the therapeutic areas. We also reviewed the relevant clinical guidelines that were cited in the CCE assessments.
We found that the mean frequency of RWE was 3.8 per ICER scoping document, 0.7 per drug per ICER CCE assessment, and 1.6 per drug per clinical guideline. In the ICER scoping documents, RWE was most frequently used to inform the outcome (55%), followed by the population (20%) (Figure).
To inform the effectiveness, safety, and treatment patterns of the drugs, 53%, 44% and 3% of RWE were used in ICER CCE assessments, and 41%, 30% and 39% were used in the relevant clinical guidelines, respectively. When stratified by disease areas, our findings indicated that RWE was used least in the area of oncology, compared to the other areas. Additionally, we also found a positive association between the time a drug has been on the market and RWE use.
Why is RWE infrequently used to inform drug-specific clinical outcomes?
RWE has a minor role in informing drug-specific clinical outcomes such as effectiveness, safety, and drug discontinuation in ICER’s value assessment process. Main reasons for the slow uptake include payers’ concern about the quality of RWE and their lack of ability to evaluate RWE, implying an overall lack of awareness and underutilization of quality assessment tools for RWE (there is one tool developed by the CHOICers!) Another reason is the time required to generate RWE. ICER reports are usually published near the time drugs are approved, while generating RWE requires a drug to be on the market at least sometime. Our second study found that a drug that has existed on the market for a longer period of time may produce more available RWE to inform its clinical benefits and harms. ICER often updates the assessments and incorporates recent RWE that was not available originally, which can sometimes result in a significant difference in conclusions. Our finding suggests that there may be a need for reassessment of a drug and adjustment of health care decisions based on updated evidence after the launch as more RWE becomes available. Finally, our study indicated that RWE was rarely used in the setting of oncology. This finding suggests that the assessment of oncology drugs still relies heavily on trials. Perhaps RWE does not fit into the current framework of the regulatory process for cancer drug development. Also, there may be a lack of incentive to conduct post-marketing research to assess RWE after FDA approval.
There are few pre-registered RWE. Is this a problem? What is being done to encourage registration?
The infrequent preregistration of RWE is an issue. Preregistration can improve reliability and transparency of RWE which become increasingly important with the growing interest in the use of RWE. Lack of transparency regarding how evidence is generated using secondary data has been one of the major barriers to using RWE for high-stakes decisions. However, existing study registries such as ENCePP/EU-PAS and ClinicalTrials.gov are mostly oriented towards (randomized) clinical trials where data are prospectively collected and lack many of the features that are relevant for studies performed on existing data (e.g., insurance claims and electronic health records). There have been several efforts to facilitate the registration of observational studies, such as a certification process for payers and the RWE Transparency Initiative to establish a culture of RWE transparency. The initiative’s steering committee, in its recent white paper, outlined an approach designed to facilitate the registration of observational studies, particularly those evaluating effectiveness or safety (i.e., hypothesis evaluation treatment effect (HETE) study). Key steps being suggested for RWE transparency initiative are identifying a location for registration, determining what a good registration process entails, and providing incentives for routine pre-registration for HETE studies.
Why is the use of RWE by ICER increasing in economic valuation over time? Was there any specific action?
The increasing proportion of model inputs informed by RWE is in part explained by the evolution of ICER’s value framework. While the essential valuation criteria did not change, there have been updates in the economic model framework that may have affected the use of RWE. The most notable change was the use of a societal perspective for CEAs in 2017. Since this change requires models to have non-healthcare sector benefits and costs (e.g., productivity costs) that are hard to be informed by RCTs, this change might have increased the number of RWE uses.
Main takeaway
We found limited use of RWE to inform drug-specific effectiveness and safety in the US despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness. While some of the barriers to using RWE are inherent to the lack of data available at the time of drug approval, there are definitely several actions that can be taken to improve the use of RWE as we mentioned above. The gaps we identified in this study regarding what has been called for and what has been done in the use of RWE will give directions to the better use of RWE in HTAs.
Incremental Thoughts 
