Does Real-World Evidence Play a Role in the US HTA?

By Woojung Lee and Boshen Jiao

Why do we want to know how real-world evidence (RWE) is being used in health technology assessment (HTA) and what is RWE anyway?

Based on the FDA’s definition, RWE is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from the analysis of real-world data. Real-world data is defined as data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.

There has been a growing interest in incorporating RWE into the HTA process to assess the clinical outcomes and economic value of drugs, mainly due to the limited availability of evidence from randomized clinical trials (RCTs) and the desirable property of RWE in reflecting outcomes in the real-world setting.  Despite RCTs being the gold standard, long-term drug effects and economic outcomes are often challenging to collect from the RCTs.  The US FDA’s recently released guidance in 2018 on how drug companies may communicate health care economic information to payers and formulary committees that enable the use of off-label evidence related to approved indications also likely increased the use of RWE in economic analysis. Furthermore, US payers increasingly recognize the value of RWE in informing healthcare decision making, believing that RWE, while not a replacement for RCTs, can provide the best available evidence for a HTA in situations when well-controlled clinical trials are difficult to implement.

However, it remains unclear how RWE can be incorporated into the HTA of pharmaceuticals in the US (there has been a few studies in Europe though!). We thought that a quantitative examination of RWE use in the US may give US payers an objective view of the possible roles that RWE can play in the HTA of drugs. This is especially important considering that the use of RWE varies from payer to payer and is generally limited because of the lack of understanding of how to incorporate RWE into the value assessment process. A better understanding of the current roles of RWE can help facilitate its use in the HTA.

What did we do?

We conducted two studies.  In the first study, we assessed the use of RWE in the economic assessment of drugs by the Institute for Clinical and Economic Review (ICER), a nonprofit independent research organization that evaluates the clinical and economic values of medical interventions. Their reports have been increasingly used by the US payers in their decision-making processes. Specifically, we reviewed the long-term cost-effectiveness analysis (CEA) and potential budget impact analysis (BIA) sections of final evidence reports published by the ICER between January 2014 and June 2019 and extracted evidence of RWE use. 

 We identified 407 RWE uses in CEA and BIA sections of the ICER reports, which accounted for 33% of all model inputs (i.e. 67% of model inputs were not informed by RWE). However, this proportion ranged from 4% – 77% for each final evidence report, showing a large variance. (Figure)

We found that over time (from 2014-2019) there is an increase in the use of RWE to inform model inputs. The most common reasons were to inform mortality or disease progression rate (29%) and health care costs (21%) (Figure).

Drug-specific clinical inputs such as drug effectiveness, drug-specific discontinuation rate, and adverse drug events were rarely informed by RWE (<3%). The most frequently used study design was a retrospective cohort (50%) followed by a prospective cohort study (17%). In terms of the data sources, registry data were the most frequently used data (40%), followed by administrative claims data (18%) and patient survey/diary data (18%).

Nondrug-specific clinical inputs (e.g., disease progression and mortality rate, patient characteristics, and incidence) and economic inputs (e.g., health care costs and treatment pattern) were mostly informed by retrospective studies whereas drug-specific clinical inputs (e.g., drug effectiveness and adverse drug events) were more likely to be informed by prospective studies. In terms of the data sources, registry data and administrative claims data were the main sources of nondrug-specific clinical inputs and economic inputs. Drug-specific clinical inputs were informed by diverse sources including electronic health records, claims, as well as registry data. Not surprisingly, only 1% of RWE were pre-registered (one meta-analysis and 4 prospective studies). We also found that about 30% of RWE was sponsored by industry.

In the second study, we assessed the use of RWE in ICER’s scoping and comparative clinical effectiveness (CCE) assessments. We examined the frequency of use, trends, and reasons of RWE use overall and further stratified by the therapeutic areas. We also reviewed the relevant clinical guidelines that were cited in the CCE assessments.

We found that the mean frequency of RWE was 3.8 per ICER scoping document, 0.7 per drug per ICER CCE assessment, and 1.6 per drug per clinical guideline. In the ICER scoping documents, RWE was most frequently used to inform the outcome (55%), followed by the population (20%) (Figure).

To inform the effectiveness, safety, and treatment patterns of the drugs, 53%, 44% and 3% of RWE were used in ICER CCE assessments, and 41%, 30% and 39% were used in the relevant clinical guidelines, respectively. When stratified by disease areas, our findings indicated that RWE was used least in the area of oncology, compared to the other areas. Additionally, we also found a positive association between the time a drug has been on the market and RWE use.

Why is RWE infrequently used to inform drug-specific clinical outcomes?

RWE has a minor role in informing drug-specific clinical outcomes such as effectiveness, safety, and drug discontinuation in ICER’s value assessment process. Main reasons for the slow uptake include payers’ concern about the quality of RWE and their lack of ability to evaluate RWE, implying an overall lack of awareness and underutilization of quality assessment tools for RWE (there is one tool developed by the CHOICers!)  Another reason is the time required to generate RWE. ICER reports are usually published near the time drugs are approved, while generating RWE requires a drug to be on the market at least sometime. Our second study found that a drug that has existed on the market for a longer period of time may produce more available RWE to inform its clinical benefits and harms. ICER often updates the assessments and incorporates recent RWE that was not available originally, which can sometimes result in a significant difference in conclusions. Our finding suggests that there may be a need for reassessment of a drug and adjustment of health care decisions based on updated evidence after the launch as more RWE becomes available. Finally, our study indicated that RWE was rarely used in the setting of oncology. This finding suggests that the assessment of oncology drugs still relies heavily on trials. Perhaps RWE does not fit into the current framework of the regulatory process for cancer drug development. Also, there may be a lack of incentive to conduct post-marketing research to assess RWE after FDA approval.

There are few pre-registered RWE. Is this a problem? What is being done to encourage registration?

The infrequent preregistration of RWE is an issue. Preregistration can improve reliability and transparency of RWE which become increasingly important with the growing interest in the use of RWE. Lack of transparency regarding how evidence is generated using secondary data has been one of the major barriers to using RWE for high-stakes decisions. However, existing study registries such as ENCePP/EU-PAS and are mostly oriented towards (randomized) clinical trials where data are prospectively collected and lack many of the features that are relevant for studies performed on existing data (e.g., insurance claims and electronic health records). There have been several efforts to facilitate the registration of observational studies, such as a certification process for payers and the RWE Transparency Initiative to establish a culture of RWE transparency. The initiative’s steering committee, in its recent white paper, outlined an approach designed to facilitate the registration of observational studies, particularly those evaluating effectiveness or safety (i.e., hypothesis evaluation treatment effect (HETE) study). Key steps being suggested for RWE transparency initiative are identifying a location for registration, determining what a good registration process entails, and providing incentives for routine pre-registration for HETE studies.

Why is the use of RWE by ICER increasing in economic valuation over time? Was there any specific action?

The increasing proportion of model inputs informed by RWE is in part explained by the evolution of ICER’s value framework. While the essential valuation criteria did not change, there have been updates in the economic model framework that may have affected the use of RWE. The most notable change was the use of a societal perspective for CEAs in 2017. Since this change requires models to have non-healthcare sector benefits and costs (e.g., productivity costs) that are hard to be informed by RCTs, this change might have increased the number of RWE uses.

Main takeaway

We found limited use of RWE to inform drug-specific effectiveness and safety in the US despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness.  While some of the barriers to using RWE are inherent to the lack of data available at the time of drug approval, there are definitely several actions that can be taken to improve the use of RWE as we mentioned above. The gaps we identified in this study regarding what has been called for and what has been done in the use of RWE will give directions to the better use of RWE in HTAs.

Choosing between a managed care residency and a fellowship: the struggles of a pharmacy student

By Tae J. Park

I’ll set the scene for you.

Maybe it will sound similar to what you (a current pharmacy student) are grappling with. You made it into pharmacy school. You’re at an elite academic institution with a renowned medical center and your professors and preceptors are the best of the best. They equipped you with more clinical pearls than you asked for and now you are off to a great start in your pharmacy career. There seems to be a big push at your school to pursue a clinical residency, and most students seem to be riding that wave. But only after memorizing the top 200 brand/generic drugs and learning which antibiotics cover Pseudomonas do you realize maybe clinical pharmacy isn’t really for you… so what now?

It’s a struggle that I, and a handful of my classmates, encountered about halfway through pharmacy school.  Luckily, there’s a whole different side to post-graduate opportunities: managed care residencies and fellowships. To the pondering student looking for other avenues, let me break it down for you.

What is a managed care residency?

Managed care can have a relatively loose definition depending on whom you ask. In this context, managed care refers to insurance companies and pharmacy benefit managers (PBMs). As such, a managed care residency is a post-graduate program taking place at these types of companies. These programs are typically 1 year.

What does a managed care resident do?

Insurance companies and PBMs are often very large companies, with many different departments dedicated to managing pharmacy insurance benefits. Accordingly, a managed care resident can expect to dip his/her feet in many of these departments throughout the residency in blocks, similar to rotation blocks during school. Some examples of departments/blocks include formulary management, drug utilization review and clinical programs. Post-residency, many people end up working in one of these specific departments. Some managed care residents may also end up working in administrative roles within hospital systems or health maintenance organizations (i.e. formulary management at Kaiser), or even switch over to an industry role that is focused on negotiating with managed care companies.

What is a fellowship?

A fellowship is a broad umbrella term for a post-graduate program that usually takes place in the pharmaceutical industry (AKA drug manufacturers). But this is not always the case! There are also fellowships in academia and fellowships that are a hybrid of academia and the pharmaceutical industry, like the UW CHOICE fellowships. Fellowships are typically 2 years, although some are 1 year.

What does a fellow do?

Unlike managed care residencies, fellowships are usually offered by a specific department of a pharmaceutical company. A fellowship applicant can apply to a specific department, such as clinical development, regulatory affairs, medical communications, market access, or health economics. Depending on the company, departments can have slightly different names and some departments may have the term “managed care” in their titles (i.e. managed care medical affairs). This can be a bit confusing, but these are still departments within pharmaceutical companies and not payers. Rather, these departments are focused on communicating with managed care organizations for reimbursement and coverage purposes. Knowing which fellowship to apply to requires the applicant to have some prior knowledge of what department he/she is interested in.

In the case of academic fellowships, they are often specialized in a certain field of study and tied to an academic institution with a medical center. These fellowships are focused on conducting original research for publication and guiding treatment decisions. Some fellowships in academia also involve didactic coursework that confers a degree upon completion. As mentioned previously, some fellowships are purely academic and others are a hybrid of academia and industry. The UW CHOICE fellowships are a great example of this, where fellows complete coursework and an academic thesis to earn an MS degree during the 1st year and then transition to industry during the 2nd year. It is important to note that not all fellowships require the applicant to have a pharmacy degree. Some fellowships are open to people with PhDs, MDs and other science degrees! It all depends on the department and what type of skill set they are looking for. From my experience, it seems like the fellowships that are focused on research & development are often open to applicants from different educational backgrounds.

How do you decide between a managed care residency or a fellowship?

If you’ve read this far, you may have noticed that managed care residencies and fellowships are really quite different. They take place in different settings, are focused on different things, and have different structures. One thing I have noticed is that many pharmacy schools and student organizations tend to lump them together and tell the students, “hey, all these non-clinical opportunities are over there” without going into much detail about what’s what.

So here’s the news: it takes some effort on your part to decide what you want to pursue. Sounds cliché? Totally. But there isn’t much of a way around it.

If you see a summer internship at an insurance company or a pharmaceutical company, apply! Get your feet wet and see what it’s like on the inside. Didn’t get an internship or missed the deadline to apply? That’s okay, too! See if your school offers rotation blocks in these settings. Didn’t get the rotation you wanted? You’re still in luck. Spend some time to reach out to people who have worked in the field or at specific departments and ask them about their experience. Set up a one-on-one call or chat over coffee. Does it sound appealing to you? Then maybe it’s for you. Didn’t reach out to anyone? Well then now it’s really on you. There’s no free lunch.

Let me dispel a common misconception: having a managed care internship or a pharmaceutical industry internship on your CV IS NOT a prerequisite for landing a residency or fellowship. Your ability to convey that you have spent the time and effort to get to know the field and how your skills and interests match that field is more important.

My own experience…

A lot of my own experiences served as inspiration for what you just read.  I knew during my second year in pharmacy school that clinical pharmacy probably wasn’t for me. So, I started exploring elsewhere. I joined AMCP and went to their events to meet and talk to alumni who had pursued non-clinical routes in managed care and industry. I also participated in the Pharmacy & Therapeutics competition. When it came time for my teammates to split up tasks, nobody wanted to do the economic portion. It was difficult and economics seemed like a different language. Cost-effectiveness huh? What in the world is an ICER? It all seemed like stuff that was never taught in school. Nevertheless, I decided to take on the economic portion and unexpectedly enjoyed it. I learned so much while creating and interpreting a budget impact model. I never knew Excel could do so many things and it was actually pretty cool. I learned that entire departments in pharmaceutical companies were dedicated to health economics and outcomes research (HEOR), and I became curious.

I tried applying to HEOR internships two years in a row. I never managed to get one. Nonetheless, my curiosity was piqued so I didn’t stop there. I was able to find an internship at a local health plan. I took on projects that were more economic-focused and performed many cost-effectiveness analyses for the health plan. When it came time to pick rotation sites, I saw that my school offered a research block with our health economics professor. So I took that block. I was the only student in my year to choose that block. And it was awesome! I didn’t have to wear scrubs and got to work in an office setting with an endless supply of coffee. I learned new skills that weren’t taught in class, such as creating different types of economic models and using new computer programs.

When it was time to start applying for fellowships, I naturally looked into HEOR fellowships. I was intimidated, knowing that I had never done an internship in HEOR while it seemed like many other applicants had. Though I did have some relevant experience in managed care, I knew that it was still distinctly different from HEOR. Both settings perform economic analyses on medications, but from very different perspectives and scales. Managed care evaluates all the different drugs that come to market on a very broad scale, while tailoring all evaluations to the health plan’s own patient population. HEOR is much more specialized in economic evaluations and analyzes the company’s specific drugs on a deeper level for internal use while taking a national or global perspective. In some ways the skills used in managed care and HEOR overlap, but the perspective from which those skills are applied and the target audiences are different. I made sure to speak to this distinction while also highlighting how my experiences in managed care would be transferable to HEOR. I ended up choosing the CHOICE fellowship because I loved its hybrid structure—1 year in academia and 1 year in industry. Shameless plug: it’s pretty great and you should apply!

So, to the pharmacy student considering a managed care residency or fellowship, I recommend that you start exploring. And try not to fixate on just finding the exact internship you want. There’s plenty of other ways to get exposure—you just need to keep an open mind.

TLDR; check out the awesome pamphlet that Kevin Li, a current pharmacy student, made!