Public Workshop: A Framework for Regulatory Use of Real-World Evidence

Just another WordPress site

Public Workshop: A Framework for Regulatory Use of Real-World Evidence

(people chatting distantly) – Good morning, everyone

I’m Mark McClellan, I’m the Director of the Duke-Margolis Center for Health Policy I’d like to welcome all of you to this morning’s workshop, or really, today’s workshop on a Framework for Regulatory Use of Real-World Evidence This is a workshop that we are convening under a cooperative agreement the FDA and we’re very pleased to see all of you here in the room and all of you who are joining us online for these very important discussions from a range of perspectives, government, academia, industry, research, patient groups and others, for a productive exchange about this new and emerging area of greater interest in developing evidence to improve the safety and effectiveness of medical treatments that are available for patients As you all know, there’s been growing interest for some time now in using so-called real-world evidence to inform key clinical and regulatory questions about the healthcare system This is due to I think a number of factors One is the increasing availability of data from the real-world clinical practice and from patients And also the availability of improving techniques for analyzing those data Patients, providers, payers are increasingly asking for more information about the effectiveness, about the value of new and existing treatments And this evidence is increasingly being influenced by what we learn in real-world settings At the same time, regulators and medical product developers are interested in identifying what can be learned from these new sources of information to help bolster the evidence around particular medical products And improve the efficiency of the clinical trial process and potentially support regulatory reviews and a range of regulatory applications The development and use of real-world data though, is not something that’s new This has been going on for decades The FDA has used real-world data in settings where traditional randomized clinical trials are not feasible or not ethical Think about applications involving very rare populations for orphan or ultra-orphan drugs or applications involving drugs where there is an urgent unmet medical need and where good information on the medical history, the course of care for patients may be available from real-world evidence is one example As another example, in 2007, some of you may remember the FDA Amendments Act established a post-market

drug safety surveillance initiative built around real-world data That initiative, the Sentinel Initiative, now grown into an into an integral part of FDA’s drug safety surveillance evidence work, is built on real-world evidence as well So the application of real-world data and real-world evidence within regulatory context has occurred, but it’s met with some limitations Limitations that you’ll hear about today that range from lack of clarity about definitions and common understandings And also, the need to better characterize data that come from real-world settings so they can be confidently used, or more confidently used, in real-world evidence setting As you’ll hear today, there’s some very important data challenges and methods challenges in applying real-world evidence and developing real-world evidence in a broader range of regulatory decisions But this in an area where there is a lot of interest and moving forward and why this meeting is so timely This is again, a reflection of the changes in data and methods technologies It’s also a reflection of bipartisan interest, as you’ll hear This in as an area where FDA’s been called upon to take more steps in collaboration with the private sector to develop better evidence, to improve medical treatments and to apply in regulatory uses And it’s an area where, because of the changes in technology, we have an opportunity to move forward So just to go over what we’re going to hear about today, we’re going to start out in a few moments with our colleagues from the FDA, who will give an overview of the agency’s perspectives on enhanced use of real-world evidence and progress implementing the real-world evidence provisions in the 21st Century Cures Act and the recently enacted Prescription Drug User Fee Act, the sixth edition We’ll then hear from my colleague, Greg Daniel, who will provide an overview of the current real-world data and real-world evidence landscape Particularly as it applies to these regulatory contexts And highlight the framework that we provided in our recently published white paper on the regulatory acceptability and potential uses of real-world evidence in regulatory uses I hope you all had a chance to see that paper It’s available for a download on our website and on the webpage associated with today’s event We’re then going to be turning to four major topics that are part of this effort to make progress on real-world evidence That starts with a session on real-world data Data is not the same thing as evidence, as the white paper makes clear And we’re going to discuss some of the issues and limitations and challenges that must be addressed in turning real-world data into fit-for-purpose real-world data for these kinds of regulatory applications We’ll then have a session on matching real-world data and real-world evidence into regulatory applications, or use cases, real-world data and methods, that are fit-for-purpose for important regulatory questions Then a session on pursuing real-world evidence development programs that can support regulatory use And this is where I think a lot of the practical challenges will also arise And again, in turning data into reliable evidence using well-understood fit-for-purpose methods Finally, we’re going to close today with a session focusing on the path forward to allow for broader synthesis of the ideas and issues that came up in today’s workshop And a chance for some of our panelists and hopefully with contributions from all of you, to discuss potential next steps to move this important area of work forward Now there are a lot of people here in the room, joining us online, collaborating, not just from Duke, not just from FDA, not just our working group partners, who are focused on real-world evidence development And we’re going to try to give you a good sense of those perspectives today The lineup of speakers that we’ve assembled have made significant contributions to the background work that went into getting us to this point And significant contributions to the agenda, the scope of topics, that we’ll be discussing So we’re hoping for a very rich discussion in pushing this challenging, but very important set of real-world evidence issues forward Before we get started, I just want to mention a couple of housekeeping notes As you’ll note in the agenda, each session will begin by some brief remarks followed by panel discussion We also have time set aside for broader discussion, with all of you who are here in the room For those in attendance, we have roving mics for use throughout the day

We also to hope to hear from those of you who are joining us online For those of you in the room who don’t know it already, this is a public meeting, everything that you say is going to be part of the public record to try to move these issues forward And particularly for those of you who are joining us on the web, we encourage you to participate in today’s discussion in a couple of ways If you do have any potential questions for a panel member or for a discussion in one of these sessions, please send them to [email protected] [email protected] and our staff will work to get them to us to incorporate in the discussion And if you’d also like to join the discussion on Twitter, and that applies to all of you in the room, I think, too, please us @dukemargolis, the @dukemargolis handle, and the #RWE, so #RWE For those of you who are in the room, we have coffee and beverages located just outside Lunch is going to be on your own, we’re going to have an hour break just after noon There are plenty of restaurants close by and finally, I want to give you a reminder that this meeting is being convened under a cooperative agreement with the FDA, but this is not a federal advisory committee There are not going to be any votes This meeting is going to be a success if there is an open exchange of a lot of ideas on these important topics To get us started, I’d like to turn this over to one of the leaders from FDA on this topic, Rich Moscicki is the Deputy Center Director for Science Operations at the Center for Drug Evaluation and Research at the US FDA And he’s going to start out with some opening comments Rich, thanks for joining us – Thank you, Mark Always a great pleasure And I want to start out by thanking, in fact, the Duke-Margolis Center for Health Policy Especially Morgan Romine and Gregory Daniel for bringing this meeting together today I have had the pleasure of following Mark in several of these Duke-Margolis convening meetings And it usually means that my talk is reduced to, “Hello.” Since Mark usually covers everything I was going to cover So knowing that repetition is important in adult learning, I’ll let you hear some of it again I also want to take the opportunity to welcome our panelists, the audience and those of you who are watching on the webcast This is maybe the sixth, seventh, eighth, meeting I’ve personally attended on real-world evidence over the past few years So what’s different? Well, what’s different is that this is a very important step in the process for FDA As we move forward with the commitments that we now have towards real-world evidence, and its use in our own context, this becomes a critical point in that discussion Where we move from the usual public discourse into the actual process of getting real traction on using this within the agency itself So we know that we and all of you, particularly sponsors, share a very common goal We want to make sure that we get important medications to the American public that are safe, effective, but in a timely and a cost-effective manner Now we realize the limitations that our current process has brought upon us, using traditional clinical trials And so therefore, there is a benefit to using data that’s collected as part of routine care, in the patient’s own daily life to inform us about efficacy of these treatments in what we are terming the real-world So maybe real-world is a touchy phrase, because perhaps it implies that the evidence that we’ve been working with is somehow not real But in fact, we all know that this traditional method has brought to us evidence of whether or not products do really work or not And in fact, those products have benefited millions of patients But we also know that the limitations are several fold for the traditional process, right? We know that, for example, only a small number of patients with any given disease usually can participate in such clinical trials In fact, one survey said that 16% out of 1,000 people

who were interviewed had either participated or knew of some family member who had participated in a clinical trial And that’s probably, according to other sources, an overestimation of how many people can really participate in such clinical studies For example, even in cancer, only 5% of cancer patients today participate in clinical studies, even when they’re in desperate situations So this does raise some meaningful questions about the representation of the patient populations in randomized, controlled trials Even though they do provide that clear picture of their effectiveness Time and cost have been a very big subject They have risen tremendously over the last 20 years for clinical trials In today’s era of we discuss the prices of prescription medications, that heightens our sensibility around this issue of the time and cost So it’s important that we continue to look for alternatives, at least in part to supplement this process Most importantly, though, we must look for the innovative approaches that allow us to ask more questions With high-quality data to answer those questions And to do so in a very rapid way So we recognize at FDA, the importance of expanding the tools available to us for getting this understanding of the risk and benefit of medications for the sake of the American public Now Mark told us this wasn’t entirely new And I think that’s very true I’ll just reiterate that, in fact, we use real-world evidence today on a routine basis to make regulatory decisions Over the past few years, we’ve worked closely with Harvard Pilgrim as the coordinating center to increase our capacity to use real-world data to generate evidence on safety through the Sentinel System And that relies predominantly on claims and pharmacy data We have incorporated that into the routine safety evaluations that we now conduct at FDA But we’re on the verge of moving further The HITECH Act of 2009, part of the American Recovery and Reinvestment Act, launched the widespread use of electronic health records So now, by 2015, almost 100% of hospitals and close to 90% of office based physician have electronic records So that begins to tell us that we could fill the gaps that claims and other data that we have had access to may not be able to provide us But electronic health records are not built with research in mind, they’re for daily care So not everything is ideal in our ability to just mine such data So we need to adopt this technology for that research purposes And that may not be straightforward How the data is entered, stored, defined and terms are not completely standardized To use such data for analytics presents real challenges So how do we similarly make use of also mobile technologies that now access data on patients that’s not just generated within the healthcare encounter, but actually in the very daily lives of patients, as another aspect that we need to consider as we debate and move forward with the use of such real-world evidence or data (chuckles) before we make it evidence Also, such evidence then, if it’s going to be seriously considered, requires verifiability And a minimum of bias and needs to be well-controlled And we need to have predetermined analytics This is the basis of our rules of evidence But as many are aware, the randomized trial has been the favored approach to try to do all of this So how can we act within this context to minimize bias in what has been essentially by many considered to be observational data I hope we’ll get to the discussions today that we will look at alternatives statistical and other methods to limit that bias that may not involve randomization alone Now as the leadership of FDA stated in a New England

Journal of Medicine article last year, we can also use such real-world data together with a randomized trial But there is no contradiction between the generation of real-world evidence and randomization And there are many other ways that real-world data can contribute to randomized controlled trials Now also, I think you heard from Mark and I’ll repeat it again, not every product that we approve has been based on randomized controlled trial data alone FDA has made use of data from registries, case series and expanded access in its approval of products Admittedly, most often and ultra-rare disorders, where the sheer numbers themselves make it difficult to conduct a traditional randomized controlled trial So here we are, this is a process We have some new building blocks and tools, but we still need to understand how to build quality evidence needed to support the findings of safety and efficacy that will continue to serve the needs of patients Our continued dialogue, such I hope we’ll have today, the collaboration that we will require from all of us, and demonstration projects together with scientific rigor will be needed as we move forward Today, I look forward to hearing from the many experts that we’re going to have speak with us And we will continue to seek their guidance, your guidance, to construct that framework so that we may move forward with the use of real-world evidence in our regulatory decision-making Thank you very much, thanks for being here (audience clapping) – Thanks, Rich, and as usual, he said a lot of stuff that I didn’t say to frame this meeting very effectively We’re now moving into our first session, so after this framing, we’re going to build up the framework for the regulatory use of real-world evidence We have two presentations for that First is a presentation from Jacqueline Corrigan-Curay, who is the director of the Office of Medical Policy at CDER, at FDA, who is going to provide a bit more detail on FDA perspectives and framing for this work And then Greg Daniel, from the Duke-Margolis Center is going to give a presentation on the real-world evidence landscape, key terminology, considerations for developing real-world evidence for regulatory purposes, drawing on that background paper that hopefully many of you all have seen So Jacqueline, looking forward to hearing from you, thank you – Can I have the clicker? Thank you and good morning and I want to thank the Duke-Margolis Health Policy Center for just pulling together this fantastic meeting And all our panelists who are here today to share their expertise and thank you to everyone in the room and those on the web So let me see, oh, it works, okay So I’m going to just move forward Quick disclosures, there’s nothing to disclose I’m going to talk a little bit about definitions, some goals and expectations I’ll summarize briefly our experience with real-world data and real-world evidence that Rich mentioned Some foundational activities that are already under way and how do we look forward to where we want to go So oops, I think I went forward several slides So the definitions, we defined real-world data as data related to patient health status and/or the delivery of the health care routinely collected from a variety of sources And of course, that could be electronic heath records and claims and billings, but also other sources in home use or mobile technologies And real-world evidence is the clinical evidence regarding the usage and potential benefits or risk of medical product derived from analysis of RWD So obviously, this is a regulatory definition of real-world evidence And if you had the opportunity to read the recent guidance on real-world evidence from the Center for Devices and Radiologic Health, you’ll see these definitions, so we have harmonized on those definitions And so the definitions really provide a framework for the dialogue, but I always think it’s important to articulate our goals because otherwise we may become too focused on whether this is pure RWE or not and lose sight of the reasons we care And I think this was mentioned before, but we really want to try and maximize the opportunities to have our regulatory decisions incorporate data and evidence from settings that more closely reflect clinical practice And that will increase the generalizability, increase the diversity of population and hopefully also improve some of the efficiencies So we talked about, there was some reference to some expectations, that there are some expectations for FDA under 21st Century Cures

We’re to establish a program to evaluate how we can use this evidence for the approval of a new indication or to satisfy post-approval study requirements And the definition of real-world evidence in 21st Century Cures is data regarding the usage or the potential benefits or risk of a drug derived from sources other than traditional trials And you can see we’ve incorporated this into our definitions, we’ve merely separated data for evidence so that we can have the conversation about the unique aspects of those We also have PDUFA commitments to enhance the use of real-world evidence in regulatory decision-making through conducting a public workshop, such as this, to gather input on the topics Initiate appropriate activities, pilot studies or other, oops, this is going forward To address key issues and to publish draft guidance on how real-world evidence can contribute to the assessment of safety and effectiveness in regulatory submissions So as it was referenced, we do have considerable experience with real-world evidence A lot of it is through our work with the Sentinel System, which has really become FDA’s national electric system, to look at safety data And this slide just speaks to the data that’s available to us through our data partners And I think although we want to think about efficacy today, we mention safety because it’s really not only about looking at safety, it’s about making informed decisions and generating evidence So this is just one example, when we started to get case reports of bleeding after Dabigatran was introduced into market, the question was, is it behaving differently now in clinical practice than it did in the clinical trial? So using Mini-Sentinel, which I’ll just refer to as a precursor of our Sentinel, we were able to see, no, the bleeding rates were similar to warfarin, just as they were in the trial And it’s that evidence generation that I think, that knowledge that we can now leverage further as we move forward and look at using these data for evidence generation And indeed, we’ve already published some guidance that incorporates some of our thoughts on methodology and quality and looking at data But of course, we are here to talk about efficacy And I think it was referenced that the leadership have published on this topic And I think they recognize certainly the value of this, potential value for real-world evidence in the regulatory setting, but it also has to be thoughtful about how we adapt these tools and methods of traditional trials It’s not really abandoning them, it’s bringing them forward into the new settings and being very critical to make sure we obtain valid results and minimize bias And that means a lot of emphasis on what are the methods used and what are the best methods that have been developed and validated that can be combined in the appropriate research settings? So when thinking about how you turn RWD into RWE, there’s a number of pieces that have to fit together Is the data that you’re going to be use, fit-for-use? What’s the quality of it? Does it even capture what you need it to capture? As we said, electronic health records are full of data, can you get that data out of those records? Are there particular regulatory considerations you need to think about as you’re moving out of traditional clinical trial sites and decentralize and moving into practice Data standards, how is this data captured and going to be transmitted? And then, what’s the study design that’s going to pull this all together provide the evidence that you need? I think we’re thinking about all these factors and how it can be done and I’m sure you are, too And we would just urge, as you start to think about turning real-world data into real-world evidence, you engage with FDA early on this journey So we have already started on a number of activities to help us think through the best way to do this And I’m just going to, so, stakeholder engagement, meetings such as this with Duke, are really great for us to inform our thinking about this and we know that Duke published the white paper that also will help inform our thinking We also support the Clinical Trials Transformation Initiative, and they recently issued guidance, issued guidance, sorry, recommendations on registries and how to assess those registries or design them for use of real-world evidence They are also engaging in a project on mobile clinical trials, so how do we use mobile technology to bring the clinical trial out of the traditional settings? As I said, we’re working on data standards, CDER and FDA recognize that we need standardized study data terminologies to facilitate efficacy analysis That’s already under way and will help as you’re starting to submit these data to us We’ve also published a number of guidances that I would consider the foundational guidances that tell you how to move from a paper-based clinical trial to using these electronic datas in compliance with our regulations And finally, I’m going to highlight some demonstration projects that are already under way

We talked about EHRs and this really probably why we’re sitting here today If there was not widespread adoptation, we wouldn’t have the data available to us Or potentially all the data This is a project that we’re supporting through a grant to Duke’s Clinical Research Institute It’s a HARMONY-Outcomes ancillary study It’s an ancillary study to the GlaxoSmithKline study, which is a cardiovascular outcomes study with Albiglutide And what Duke is doing is trying to assess at the clinical site, could the EHR, what was its ability to facilitate recruitment, to populate baseline characteristics, so there are efficiencies, or even to identify the clinical endpoints And I would refer you to the grand rounds, and this is available on the web, I believe this is the new website from the NIH Collaboratory, to find out a little bit more information But this is the type of information that will really inform us on how you use these data and what they can tell us I think many of you are aware that there has been incredible innovation in oncology And also, within the FDA’s Oncology Center of Excellence, there’s a lot of very innovate work going on in the real-world evidence This is a presentation at the ASCO meeting and reflects a collaboration with Flatiron Health to examine how real-world data can be used to gain insights into the safety and effectiveness of new cancer therapies Dr. Abernethy is on the first panel and perhaps can speak more to this And in June of this year, another collaboration was announced between the America Society of Clinical Oncology’s big data initiative And all of this is being done through the Oncology Center for Excellence information exchange and data transformation And they’re really looking at how we can incorporate real-world data and real-world evidence into our decisions Dr. Sean Khozin is the lead on this project The other thing, when you think of medical records, you can think of going in and extracting what’s there or you can think of trying to start at the beginning and put the right data in so that we can use it for many uses This is a project that we’re supporting, Dr. Laura Esserman, which is one source Sort of enter this data once and use it many times It’s the integration of standards based tools into the EHR to bring together health care and research So it’s another important innovative way to, sorry, this is just moving on me To try and look at how we can use the EHR for research and the demonstration is in breast cancer clinical trials And perhaps Dr. Esserman will provide us more information We talked about data standards and this is, besides the data standards that we’re developing internally, for submission and efficacy, we’ve also seen the growth of a number of networks that have real-world data available for analysis and we’re thinking about how we can expand and leverage those networks So not all of them speak the someone language, the common data models, but there are consensus based standards that perhaps could be leveraged So FDA is leading a project called Harmonization of Common Data Models for Evidence Generation And that’s with NIH’s National Center for Advancing Translational Science, the National Cancer Institute, the National Library of Medicine, and the Office of the National Coordinator for Health Information So the ultimate vision for this project would be to have all of these networks be able to talk to each other and provide data for questions And it’s really to further leverage what we have already developed And finally, I’d just like to talk about evidence generation and we are supporting, now, a trial within the Sentinel, so this is a first trial in the Sentinel System It’s a practice and patient level educational intervention to increase anticoagulation use for individuals with atrial fibrillation who are at high risk of stroke And some of the statistics on the right hand show this is an important public health question, but importantly, also for us, is going to be a pilot project that will inform future interventional studies and really help us understand how these data can be used to generate evidence So I’d just like to say, we understand that there’s just a wide range of ways that you can incorporate real-world data into your studies And it can be starting from the left and including improving efficiencies, to starting to integrate it so it contributes to a greater extent to the evidence all the way up to the observational studies in which there’s no intervention being done and it’s just extraction of that data And I think for us, we need to have continued dialogue to understand how this evidence is generated and what it can inform in the regulatory setting So looking forward, I think we want continued engagement with stakeholders to identify the key questions that FDA needs to answer to facilitate sponsor use

of real-world data and real-world evidence for regulatory decisions and that will help us provide appropriate guidance in this area We want to continue to identify knowledge gaps and support appropriate demonstration projects, which we have been doing, to facilitate the development of RWE for regulatory decisions And develop a framework and program that is of use to everyone I’d like to just acknowledge some of the folks who contributed to these slides, or whose slides I borrowed And then I’d like to close with just saying if you have questions and comments, we’ve set up a, I’m really sorry here, a mailbox for you to please submit your questions and comments to us And so thank you (audience clapping) – Thank you, Jackie Let me, well, hold on Okay, let me echo Mark’s welcome to all of you this morning It’s great to have you all with us I’m told that we have over 1,500 people on our web, so thanks to all of you for joining our session today Over the next few minutes, I’m going to, let me introduce myself I’m Greg Daniel, Deputy Director for Policy in the Duke-Margolis Center for Health Policy Over the next few minutes, I’m going to take a little bit of time reviewing the need for real-world evidence to fill some of the gaps that we talked about, as well as to present to you our framework that we’ve developed and presented in the white paper As many of you know, traditional randomized controlled trials have long been the gold standard for evidence generation related to the safety, and maybe if I talk a little bit slower, related to the safety (laughs) and efficacy of medical products And randomized controlled trials will continue to be the gold standard This is not about replacing randomized controlled trials, this is about how real-world evidence can help inform regulatory decisions and can help bring additional data and evidence to bear as decisions are being made Yes, okay, however, these studies, the randomized controlled trials are, as we all know, increasingly resource-intensive and time-intensive and often don’t provide a wide array of evidence that might be needed for patient decision-making, provider decision-making, and even payer decision-making For example, evidence on longer term outcomes or evidence on outcomes that might be more relevant to patient populations may not be developed during the clinical trial process In addition, the population’s represented in clinical trials may not necessarily reflect the patient populations using the medical products, including multiple comorbidities, different age groups and other socioeconomic status indicators in groups of patients that may not have access to trials or may not be well-represented In addition, there are gaps in our knowledge when a drug or device is approved for use And increasingly sophisticated data methods for filling those evidence gaps, through the use of routinely collected health data can be useful And that’s real-world data and real-world evidence So right now, the nation’s growing electronic health information infrastructure has enabled routine and increasingly robust collection of such data and evidence This data and evidence are often used for informing patient and provider clinical decisions And at the same time, payers and providers are moving more towards payments and reimbursement models that are increasingly reflective of value and linking payment value and real-world evidence development can support those decisions as well Regulators are, as we’ve mentioned, are increasingly poised to make better use of real-world evidence and the data and methods as the data and methods mature through other uses and applications As Rich and Jacqueline have already touched on, Congress has mandated FDA to explore the use of real-world evidence within the regulatory framework, both in the recently passed PDUFA VI commitments, as well as the 21st Century Cures Act So two separate pieces of legislation governing this so FDA really needs to do this So when assessing the real-world evidence approaches for regulatory use, it’s important to make sure that we’re all using the same definitions and a common understanding of what we mean when we say real-world data and real-world evidence

These definitions are completely overlapping with the definitions that FDA put forward And just to reiterate, real-world data is any data relating to patient health status and/or the delivery of health care that’s routinely collected from a variety of sources This can include traditional sources of real-world data that many of us think of, like electronic health records and claims data, but also it can include registries and data from mHealth technologies and wearable apps Or wearable devices, apps on your smartphone, fitness trackers, as well environmental exposures It’s almost like every day, there are new potential sources for real-world data Are they ready for research purposes? Probably not, and a lot of methods and curation do need to go into that But the sources of real-world data are increasing Then the definition for real-world evidence is essentially using research methods to develop evidence from those data sources Now real-world evidence can be developed on anything Medical products, new treatment patterns, new programs to improve quality of care But as Jacqueline highlighted, for regulatory applications specific to medical products, real-world evidence can be defined as clinical evidence regarding the use and potential benefits or risks of a medical product derived from analysis of real-world data Real-world data is not simply anecdotes based on, or real-world evidence is not anecdotes based on real-world data, but does take, as I mentioned, more data curation, standardization and methods to ensure that data we’re using are fit-for-purpose or fit for regulatory purpose As was highlighted earlier today, there have been, this is not new for FDA They’ve been using real-world evidence all along In this figure, we try to highlight some of that Over on the far left side, which is representing new drug approvals, FDA has occasionally used real-world evidence for small populations In the approval of a new drug on the market, in a rare population where it might be challenging to randomize the use of historical or external controls, have been used in those situations On the far right of that slide, we’ve heard about the Sentinel System And the Sentinel System is routinely used to help inform FDA’s decisions about safety of medical products and that’s entirely using real-world data sources and observational methods So what we’re really talking about today is what’s in the middle So all of the kinds of regulatory decisions that happen after a product is on the market New indications, labeling changes, new information to be added to the label, all of these require a regulatory decision But they all require, there’s sort of a question, how can you use real-world evidence to support those decisions? The middle is where we are focusing and we’ve put forth a framework in our paper that tries to at least start the conversation about, when you ask the question, can FDA use real-world evidence for what’s in the middle there? The best answer is, it depends So what does it depend on? And what are the kinds of considerations that you need to walk through to ensure that the types of real-world evidence that you’re putting together are appropriate for the decision at hand This is a framework that we’ve put in our white paper and it really does mirror well the considerations that Jacqueline outlined But essentially on the far left side, you have the major two considerations Number one, what’s the regulatory decision? Is it a new product on the market? Or is it a new indication or a labeling change? And how close is the approved product label that already exists, how close is that to the actual regulatory question at hand? If it’s the same population, the same endpoint, but you’re adding additional safety information to the label, or is it a completely new population? It’s a little bit further from the existing indication All of those need to be considered Then, what is the clinical context? Or area that the real-world data will be collected? Certain disease areas give rise to better data collection or more frequent encounters with patients in the healthcare setting, so give rise to potentially more complete data and a better ability to leverage real-world data sources Other clinical conditions or clinical questions might have many more challenges in terms of selection

biases that may not be well-addressed with the available observational methods or other types of issues with the clinical context that might give rise to different questions about how real-world evidence might be supported there That moves over then to the real data and methods considerations Given the regulatory question at hand and the clinical scenario that we’re talking about, can you actually collect real-world data that’s valid and reliable for that purpose? Do we have the right quality checking and data curation methods? And then finally, from a methodologic perspective itself, this gives rise to a lot of different potential applications of methods to produce real-world evidence Real-world evidence can include randomized studies Randomized pragmatic trials that are conducted in the clinical setting, but still include randomization as well as non-interventional studies or observational studies that rely on existing data and populations that are not randomized There are a range of robust and reliable analytic techniques and study designs that can be used across these types of designs and the question of, given the regulatory context and the clinical context and the availability of real-world data, what are the best designs and methods to apply to the data to get valid and reliable estimates that would be useful for regulatory decisions These interlocking considerations will guide whether or not real-world evidence approach is adequate and appropriate for the intended regulatory application And again, we’re putting this framework forward as a discussion tool to help get today’s discussion going We’d love to hear throughout the sessions, how applicable this is or how you’re viewing this We’ve built this through a planning group, an advisory group that includes folks from the manufacturer perspective, the patient perspective, the methodologic and academic perspective and we would like to get a lot of your feedback today on this, thank you (audience clapping) – Greg, thank you for framing today’s discussion and thanks to Jacqueline, as well You heard the issues that we’re going to try to cover during the course of the day You’ve seen a couple of frameworks that are useful for thinking about all these issues And now we’re going to go through these in more detail I’m very pleased to be joined on this panel, focusing on a development of fit-for-purpose real-world data with some experts in the area, from a wide range of perspectives As you heard, real-world data is not the same thing as real-world evidence But you can’t get to real-world evidence without access to and understanding of real-world data that are valid and reliable for the intended purpose For this session, we’re going to talk about some of the best practices and key considerations for developing fit-for-purpose real-world data And I think we’ll also hear from the panelists about some of the challenges in getting from where are now to where we’d like to be with fit-for-purpose real-world data We’re going to start out with some brief opening comments from our panelists and then have a bit of followup discussion and open it up to all of you as well Let me start by introducing this distinguished panel At the far end is Kevin Haynes, a clinical epidemiologist on translational research for affordability and quality at HealthCore So HealthCore is a group affiliated with Anthem that does extensive work in real-world evidence development Amy Abernethy, is the Chief Medical Officer, Chief Scientific Office and Senior Vice President for Oncology at Flatiron Health There and in here previous career at Duke, very much involved in developing richer data for reliable real-world evidence applications Sally Okun, is Vice President for Advocacy, Policy and Patient Safety at PatientsLikeMe and has led some of the leading, or has helped guide some of the leading efforts around incorporating patient data as a source of, patient-generated data, as a source of real-world evidence And Laura Esserman, who you heard referenced earlier by Jacqueline as well, is the Director of the UCSF Carol Franc Buck Breast Care Center and Professor of Surgery and Radiology as UCSF and is involved in implementing systems, as you heard, across a range of electronic health record and other data source networks I’m going to turn to Kevin to start us off – Thank you, Mark, and thank you for the opportunity to speak today I think we’re all interested in real-world data to close these gaps in data that it’s necessary to close the gaps in evidence to ultimately to close gaps in care So that’s one of the hallmarks of what

real-world evidence can do I’m part of the Sentinel System, I’m one of the Data Core Co-Leads in one of the pilot projects that you already heard about this morning with the IMPACT-AF project, is a vital interest to showcase how we can begin to close these caps in care through these interventional studies So these small pilot studies that don’t feel small, but can begin to close those gaps I think that some of the key pieces of challenges that we’ll talk about today is the fragmentation of the data and the fact that data is often held close to the institutions that create that data And because of its both perceived value as well as the piece about business risk, with regards to data sharing We talked a little bit this morning about data standards and the things that need to be done to close those gaps and how we think about protecting patient privacy, the integrity of the use and protecting the business interest of those data sources are three hallmark challenges that we, as we embark on closing these gaps of data, will need to maintain I take a philosophical view on this and take an extreme observational question to showcase the need to integrate data So if you were to ask the question, what effect does antibiotics have on the first two years of life and prostate cancer or colorectal cancer What data source, what real-world data source would you use to close that gap in evidence today? Well, it doesn’t really exist My antibiotics are in a closed pediatrician’s office somewhere in northern New Jersey My child, Luke, can’t even study this question unless all of his data still integrated from PEDSnet and PCORnet all he way to Medicare data at the other end of the spectrum So currently, I’m hoping that my grandchildren will be in a position to observationally analyze that level of real-world data I purpose two forms of data linkage There’s longitudinal data linkage People hit 65, they enter Medicare, there’s a huge disruption in the longitudinality of observational data And the second data linkage challenge is in defined periods of time, the information that’s held deep inside a clinical system, like vancomycin one gram, Q12 on five south, that might wife might be entering into an information system today at the hospital near the University of Pennsylvania, is totally unavailable to a longitudinal picture of administration claims There’s two major tenets of data linkage from a gaps perspective that I see in real-world data to close And I’ll close with HealthCore is very committed into this space, we’re actually part of PCORnet, as one of the two funded along with Humana, health plan research networks And we’re actually also contributing to another pilot project with regards to the ADAPTABLE study, which I think you’ll hear about a little bit more this afternoon, as health plans begin to engage in the space of enrolling or recruiting patients into large pragmatic clinical trials And I think we’ll hear about some of those themes as we go through the day – [Mark] Thanks, Kevin Amy, we’ll turn to you – Fantastic, as Mark said, my name is Amy Abernethy My comments as it relates to real-world data and solving this problem comes from both my observations at Duke, where we were looking at how do organize electronic health record data within the context of a single health system as well as at Flatiron Health, where we’re doing this both because we own an electronic medical record as well as trying to aggregate data from across medical records Building on Kevin’s comments, a few observations One is that the EHR as a data source provides the opportunity for a longitudinal view But it’s often just pieces of that story In order to create the story, you need to aggregate the pieces across time If the person is receiving care in a primary care setting and then goes to the oncologist clinic, if they’re in different health systems or in different clinical spaces, you need to bring those two pieces together The second is the criticality of getting data cleaned up It is boring, messy work, but it has to be done There’s no magic bullet for the messiness of electronic health record data And importantly, many of the critical data points live in unstructured documents So being able to process PDFs and pull out, for example, information such as histology or biomarker status, is something that we have to solve for if we’re going to have data that serves our research needs

The third is linkage and data linkage is, again, a tough beast We’ve shown at Flatiron that we can link, for example, genomics data and electronic health record data But as you start putting in more and more data sets, risk of re-identification goes up dramatically And we have to solve the privacy issues as well as the linkage issues And the other think is linkage is expensive Another that’s important to solve for is data and context Essentially, one of the ways that we’ve learned, both at Duke and at Flatiron to make sense of these data, is to generate patient stories that represent what happens to people across time Because individual patient stories aggregate into cohorts or into populations Being able to put data into context, you can understand when the biomarker was done and how does that lead to the choice in therapy and the choice in outcomes That then takes me to my next point, which is real-world data is nothing if we don’t include endpoints Mortality, in cancer, we care about whether or not the tumor responded or came back Patient reported outcomes, as you’ll hear from Sally So endpoints have got to be embedded in our data set, and again, not for the faint-hearted And then, finally, the importance of having data that we can trust, where we’ve documented quality and reliability, and we also have provenance and the ability to trace back to source Because if we’re going to generate these data sets, to then ultimately submit to the FDA, they’re going to want traceability to be able to ensure that the information is indeed representative of what the clinician said or what we knew happened at point of care So that’s another key feature of these data sets With that, I’ll turn it over to Sally – Thank you so much Thank you, Amy And thanks for having us here today I think what you’re going to start to see is sort of a theme A theme of gaps, a theme of needing to fill those gaps and finding different ways of being able to do that I’m going to put this in the context of the quote that you see up there The process of developing fit-for-purpose real-world data is really quite well-suited to this If you’re crossing the river by feeling the stones, it requires you to look ahead to where you want to be on the other side of that river It requires that we incrementally and strategically navigate that river bottom to find those right stones The right supports that will get us across to the other side This process is rarely linear So we may in fact have to go off course at times in order to be able to find the right path We might even have to do something with others to get us across, find other tools to make it across Some river crossings are going to require us to really sort out the ways of collaborating and coordinating what we need to do to get our plan to get to where we need to go Defined broadly, fit-for-purpose really means something that’s good enough to do the job it was intended to do or designed to do The interesting conundrum about that statement, in our discussion about real-world data, is that most real-world data are good enough for the job they were designed to do But they’re not necessarily designed for regulatory decision-making Do we need to develop new real-world data that actually has been specifically designed for this purpose of regulatory decision-making? Or might we instead preserve the integrity and the original intention and richness of real-world data and focus instead on developing fit-for-repurpose? Understanding that that will also require new tools, new methods and new understanding What does it mean for a person and patient-generated data? Certainly, a very rich and increasingly available source of real-world data, yet has been really not fully harness to its potential to be fit-for-purpose or for that matter, fit-for-repurpose So I’d like to have us consider one use case It’s fatigue in chronic illness And more specifically, fatigue in multiple sclerosis Fatigue is present in about 80 to 90% of people who are living with MS Fatigue is an individualized experience, yet it can be measured Fatigue negatively impacts a person’s daily life and can interfere with a person’s role and responsibilities in ways that are very, very challenging for people to live with on a regular basis Yet, fatigue is rarely measured in clinical encounters unless the patient themselves brings that experience to the attention of the clinician during the visit Even then, it may not be captured in the clinical record and the clinical code might not be assigned to it for billing purposes and therefore it’s not going to be found in claims data If a patient is prescribed one of the two most frequently used products for fatigue in MS, modafinil and, I’m not even going to try, you know, amantadine, the connection to fatigue will not likely be found in prescribing or claims data because the purpose for prescribing a specific product is usually not captured In fact, neither of these product labels include fatigue in MS as an indication A recent systematic review of these products used for fatigue in MS found only 11 studies with sufficient effectiveness data to be evaluated The sample sizes ranged between only 19 patients to as much as 121 patients The review found that modafinil was not

considered the more effective product Yet a quick look at the data within PatientsLikeMe, in our MS community with its nearly 60,000 members, suggests that modafinil is actually more effective in managing fatigue in MS We have over 3,000 members with experience with these drugs who have completed over 1,200 treatment evaluations for use in specifically managing fatigue These evaluations provide data on start and stop dates for the drug, including reasons for stopping, when that patient did stop the drug and told us about that Dosing and frequency, perceived effectiveness for managing the fatigue, the side effects, even though we’re not necessarily talking about safety, we have some information about the side effects people are experiencing with these off-label use and the severity The overall burden of using these products, the adherence to them, the cost out-of-pocket, and a host of information in narrative on advice and tips Developing fit-for-purpose real-world patient-generated data can begin by understanding the data sources that already exist And evaluating its capacity to be repurposed for regulatory decision-making Fatigue represents a commonly experienced and often debilitating symptom across many chronic health conditions My example is just one condition in which it can and is being measured and is sufficiently important to assess as an efficacy endpoint In addition to the traditional outcome measures in a range of chronic conditions These data are not likely to stand alone The journey to the other side of the river will require additional tools and collaboration with other data holders to compliment and in some cases help them complete their journey toward fit-for-purpose of the real-world data that they have in their databases So it’s time to harness the potential of developing fit-for-repurposing patient-generated real-world data for use in regulatory decision-making The challenge now is feeling for and finding the best stones and tools to get that data across to the other side of the river Thanks so much – [Mark] Thanks, Sally, Laura? – Thanks very much for including me, Mark Our group is really trying to refocus clinical practice on high-quality data collection so that we can transform the point of care into a patient-centric data hub where learning and improvement are actually a part of the routine of care As a surgeon, I can’t help but think of a surgical analogy, which is first, stop the bleeding I think that what we’ve heard is that all these problems with data really stem from basically the way we practice medicine We’ve been practicing the same way for 70 years Maybe it’s time for a change If you harness a couple of the ideas of quality at the source, enter once, use many, and you put this into medicine and really start thinking about taking back medicine to enable the clinicians to do what they went into medicine to do in the first place So if you actually, again, not that you know every piece of information, and you don’t, and you can always go back and try and get it But actually, in a lot of conditions, we know the basic building blocks We know the key pieces of information And they can be assembled into these simple checklists That’s what we call the OneSource Checklists And it’s not just clinician data, it’s patient data Really, I don’t think we should ask clinicians when a patient went into menopause Actually, we should ask the patient There’s a lot of things we ask that should come directly from patients So the idea is if you actually have that data and you can use it in real time, you would enable a host of downstream improvements You would enable quality improvement, cost transparency, trade-offs, decision support, trial matching An ability for data to seamlessly flow into trials And I think that another key principle is this idea of integrating research and care We always talk about, oh, well, there’s care and then there’s research Really why are we practicing medicine? We should be practicing to improve But you can’t improve if you don’t know what your outcomes are and you don’t know what things cost and you don’t know what patients opinions are about it No business would ever run that way But unfortunately, medicine is largely in the reimbursement business, I’m sorry to say And I think we just need to get it back so we need different systems In terms of safety, I think we have to have a lot more discipline around adverse events You can find it In running the I-SPY trial, one of the things that we don’t have is we don’t have the adverse events built in A lot of the thing that are reported to the FDA are reported in the severe events of adverse events reports But they’re not really in the record

How ridiculous is that? Why aren’t we tracking that? And another part of the problem with electronic records is they were built to mimic the way paper records were built That means everybody creates their own record There’s no notion of a shared record Amongst clinicians, let alone clinicians and patients And if you have an adverse event, you don’t need 25 different versions of that event You need one and you need some discipline So that means, actually that we can’t just fix these problems from the back end Someone, somewhere, we have to go to the front end and say it’s time to change practice That means mindsets, skillsets and tool sets have to be changed And another big problem is that the electronic health records are really institution-centered And not patient-centered and that’s another big problem And it will bring up problems with governance, which I do not know how to solve But it is actually a really important problem We are trying to work across the UC system to makes this happen To make this concept a reality I think if we can get a big organization to adopt this approach, and it’s not just breast cancer, it’s just a good example, but you could do it in breast, prostate, ortho, doesn’t matter, orthopedics It shouldn’t matter, right? And it’s going to take clinician leadership and patient leadership And I’d like to say just another two quick points And that is the way in which we conduct trials today can also help us merge fields and change We’re conducting two big trials, one is a PCORI-funded trial to look at how to personalize screening And we are actually designing that trial from the get go to be institution independent and build in the adverse events and all of the feedback in real time with payers at the table and guideline makers at the table with the idea that when the data comes in, it will change practice in real time And getting payers and people to adopt up front, I would not say it’s an easy journey but we will get there Another opportunity, I think, is running the I-SPY trial, which is looking at people of the highest risk cancers and trying to think about can we use early endpoints to identify new drugs that work? Well, we’re at this critical juncture now, where some of our treatments have improved the chance of a complete response three-fold I ask you, why should I give anybody the standard of care? Adriamycin, which patients call Red Death, would not graduate from I-SPY Why should I be giving them that drug? So I think actually when you have been running a platform trial for years, seven years now, and I have really tight data on what the standard of care is and I have biomarkers that allow us categorize the tumor types and know what their outcomes are, why do I need to keep assigning the standard of care? Why can’t trials and platform trials emerge and evolve into real-world evidence generators And if you take your early endpoints, part of the reason why the early endpoints haven’t worked is people don’t keep them, people who have these endpoints, all the way to the end And that’s actually how you prove it Your early accelerated approval would allow you to say, okay, I’ve got a good response And that long-term follow up actually should be your final approval Using a base of historic data that is tightly designed I think it’s an interesting opportunity to think about how we evolve the trial system Because Lord knows, the confirmatory phase III trials with eight to 10,000 patients is not working It’s a waste of money, it’s not helping us, we have to change I think we have to broaden our ideas of what is real-world evidence, but it has to start by transforming the clinical care process We should not be practicing with the SOAP note anymore It’s time to move on – Yeah, well, what you’ve laid out is not only about transforming the clinical trial process, but transforming care so that both are based on– – Correct! And by the way, who needs data most? The clinicians who are taking care of you! – That’s right, that’s right And I’d say for all four of these presentations, it’s nice to see both that the passion and the practical side, so you can see a vision here of how real-world evidence driven by, at its core, reliable, timely access to needed data Use once, or sorry, collected once, used many times, could really transform clinical trials, evidence development and especially patient care

As you’ve heard, we’re not there, yet And I want to go back to some of the, a little bit more discussion of some of the key reasons that you raised I’d like to start with some of the challenges, just around the complexity and reliability or lack thereof of the data itself And then turn to some of the institutional or maybe business interests Kevin, I think you said proprietary issues that may be in play here And how particularly this additional FDA effort around regulatory uses of real-world evidence can help transform all that As well as the data can help the regulatory evidence succeed So let’s start with some of the data challenges that the data itself that you all mentioned I heard over and over again that healthcare data is complex and messy and hard to collect consistently And that goes to some of the comments that you all made about free text entries, electronic data that’s based on an electronic version of a paper record I know from working with Rich Platt and Sentinel, there are, I think, Rich, something like 100 ways to report on a blood sugar hemoglobin AIC test, and some of what you all described was, I don’t think maybe brute force is not the right word, but just hammering these data into standards through some combination of developing algorithms that can take all of these different formats and turn it into a data element that’s cleaned and more reliable Laura, you mentioned developing lists that could expand that over time, checklists, that start with maybe a limited amount of key data elements but grow, and structured – That are structured – And grow from there So that may be one path for it I’d like to hear people thoughts about how to address that I’d also like some thoughts about some of the ideas that you see from other Silicon Valley colleagues who are very interested or talking a lot about about data lakes and freeform text and using additional tools to take those messy data and nonetheless, turn it into something that’s like, like you said, Amy, kind of a story for a patient Even though we don’t have standardization on the underlying elements, at least we can get to effectively, perhaps, effectively standardization on the stuff, the results that matters That might be, I don’t know if that’s another really feasible approach I don’t know if these visions for data lakes are really souped-up data swamps, but I’d appreciate some further thoughts on how we can actually, we’ll get to the business case and the governance and things like that in a minute But just, how we can go faster It seems like a pretty daunting task given the complexity of healthcare data – A couple of comments here This is where we need to learn from the tech industry or actually have the tech industry help us solve this problem A couple of key points, I like the language you used about hammering into standards The truth is, what we need to do is systematically go through and clean things up When you’re talking about Rich’s example with the many definitions of glucose, getting to single concepts that have single sets of units that we can analyze One of the things that we’ve learned at Flatiron is that as you start to develop the process, you reuse that process many times So you learn how to get albumin into a single concept, you can now reuse that for glucose, you can use it now for another set of biomarkers, et cetera So one is, that process of developing or hammering out the standards can be used and reused itself, if you take some of the core principles of building algorithms, technology, and putting that together But the second part that you brought up, and you were talking about the issue of essentially aggregating through data lakes, or swamps, there is no magic there You still have to have a system to which you’re going to make sense of it But once you do make sense of it, one of the things I can show you is how quickly you can then start drawing insights from the data One of the ways that we focused on making sense of it, within the context of Flatiron as well as at Duke, has been by using patient stories Because if you can think about a person with lung cancer and know in general this is how it looks when the person gets diagnosed at stage II, when they have their surgery and radiation, when the disease ultimately comes back and you do a set of biomarkers that then dictate chemotherapy And then potentially hospitalization or death, you can now start to generate a set of data stories that allow you to link data together in meaningful context that readies for analysis The last two pieces I want to hit on, one is that we have a belief system that data need to be perfect

And in fact, they don’t – They don’t – We’ve been working with claims for 25 years and we know how to work through claims We have same kinds of things we need to work through with electronic health record data We can do simulations to test, when do you need really highly reliable data and when can you get away with less reliable data? But more importantly, one of the things that we’ve learned over and over again is the more that you work with data and analyze it, the cleaner the overall system of data gets Because the two pieces reinforce That’s my last point, which is that ultimately, the analyses, you can just wait for the data to be perfect before you do the analysis Because you do the analysis, you compare it to what you would have expected to get, for many core analyses, and then you use that information to update the overall general set of data – I’d like to maybe make a comment You think about the transcontinental railroad This is where we are now And these are the efforts that we have to make to try and make things work We should take those learnings and say, honestly, how do we want to see the practice of medicine in the future? Truly, we’re not going to be able to screen scrape our way into a different future, partly because you’ve got to change the mindset of the clinician By saying, okay, you don’t have to change anything you do, just keep doing what you’re doing and we’ll just fix it all on the back end That’s of course what you have to do today But really what you want people to think about is oh, what are, do I need albumin? Do I need hemoglobin A1C? Okay, just build it into what I’m thinking about, but get people receptive and ready to say, oh, how many people did I take to the OR last month? How many times did I take them back for re-excisions? Getting people to start– – Is that where your checklist comes from, you start with these clinical questions? – Absolutely – And then get them to help you generate the reliable data elements? – Yes, we have been able to generate the checklists and people can agree on what the key data elements are In fact, when we brought epidemiologists, all the scientists together and all the clinicians together, even regardless of specialty, you want to know what? Everyone needs the same data It’s actually not some great surprise The trick is trying to turn them into structured lists and actually figure out a framework where people can actually use it in clinical practice And the reason it’s so hard is means you have to totally re-engineer the way you run your practice and think about, it’s actually quite, it is feasible, we’re doing it And actually, I have to say that the current way electronic records work is the reason why we’ll be able to fix it Because people are suffering with the burden of documentation And I actually have that to thank for enabling change because I think the relief you’ll get, plus all the things that you’ll be able to do with it You gotta get people excited about the opportunities and say, wow, what could I get back? What could I learn? I want to be a part of the process of learning We have to step it up here on a clinician side and say that, yes, we can do this and yes, we can make a change so that the future can be brighter That we learn about all of these problems and solve it I would say, to me, the biggest problem is going to be governance Because the problem is, where does the source of data lie? Who governs that? Right now, it’s been convenient that institutions do it or companies do it But people get their care, as you said, Kevin and Sally, across all kinds of places So where is that right central repository? Or how is it that everyone has to make their data open and searchable These are really important and difficult issues – They are, I want to get to, just to pick up on this point about are there key data elements that in a particular, keeping with the framework that we talked about earlier, a particular fit-for-purpose particular use case, that could be agreed upon And maybe, Sally, you might go back to your example of multiple sclerosis You’ve done a lot of checklists in the context of breast cancer and other contexts, but for multiple sclerosis, for generating real-world data, do you see a path that could include? You’re familiar with those patients, that could include some of the key clinical data that Laura was describing, but also key patient-reported data like fatigue – Absolutely, and for those who are not aware, PatientsLikeMe is moving into a whole new realm of data in terms of thinking about integrating new sources of data through our DigitalMe initiative Which is actually collecting bio-specimens from our patients and MS is one of the conditions within which we’re doing that What we’ve done is we’re parametrized that entire experience of living with MS and we start to understand what are the state changes that people go through over time? And how can we begin to biologically understand what’s happening at that point of that state change And patients actually putting their hand up and saying, “I had a flare and I want to “better understand what’s going on,” or, “I had a relapse.”

So the opportunity for us to suggest that the phenotypic, the phenomics of the experience is something we’ve been at now for 12 years and have been collecting in a very codified way In ways that, actually, the FDA has become much more aware and understanding of, given our research collaboration with them over the last few years that it’s not so different from what some of the data is that they’re already accustomed to, when it’s collected, curated and coded appropriately for use in other settings beyond what we have I think the other piece that we feel is really critical is that the source here is the person If there were going to be a governance, I’d say it’s the patient I’d say it’s the person And we have to find a way of being able to have someone, whether it’s you and your son going in and trying to determine an antibiotic use or myself going in now and closing in on becoming a Medicare beneficiary, to say I want to know that these are the things are important to me at this age my life And here’s the things that have happened to me in the past, and how have we learned from that? So I can have, going to my older age feeling like I’m confident that I’m going to access to the type of care, the interventions, and the total whole health that I would expect at this stage of my life – I’d love a comment from you, too, Kevin Go ahead on this, then I want to push on the data linkage issues, so even if you have the list of elements that you’d like to put together, how we do it? – It requires others – Yeah, so to take and build on the fatigue example, I think claims data is longitudinal, but delayed, right? So it does set itself up for good, robust environments to do some of these safety analyses that we’ve talked about Clinical data is deep, but not complete Because you don’t have the information that may have gone last week across the street I’m still on Vicodin because it’s active med in my EMR So I don’t want to get that transferred all over the place, because I’m not And then patient data is complete, but it’s not standardized and it’s not codified I think all three, payers, patients and clinicians have a stake and I think that there’s going to be the demand on the change that, if I as a payer, knew that the patient was in fatigue, how can I provide better care for that patient? Maybe I send them a Lyft to help them get to their appointment Maybe I provide an additional services to them to add value to them But I don’t know that unless the patient tells me that or is using some app that may only be at UCSF that then I can’t see that data and provide action on When we as a health system close those gaps in data, then we as researchers and evidence builders will have the available data and be able to draw inference and evidence out of that The day supply field in pharmacy data used to be really bad because it wasn’t audited, it was where the pharmacist’s hands hit the keyboard Once it became audited, it became really, really clean And it helps Sentinel immensely at being able to define episodes of care and those types of things So sometimes when you drive things into the different payment models, you’re going to improve the data environment If we improve the data environment, we will undoubtedly be able to improve the evidence that comes out of that data environment – So for sure, we pay people for doing 12 point review of systems If you paid them for completing a checklist, or for filling out the adverse events, things would change overnight If you said, oh, I’ll do accelerated approval And I, the FDA, will make it available if you fill out adverse event report forms, that’s safety And payers said, oh, and I’ll pay for it, only under coverage with regulatory evidence Meaning, you have to fill out the outcomes You would get the data you need and things would change overnight, because people do what they’re paid to do – I see Kevin nodding, but Amy, your thoughts again? – I’m more dubious (laughs) We studied this at length at Duke One of the things that we asked was, if we build systems and force people to put information into systems, how many data points could you get people to structure on the front end before things fall apart? It turns out that it is highly related to your seniority in the system That much is for sure But the answer was somewhere between three and seven data elements per visit So you gotta decide exactly what three to seven data elements were Why, because even if you fixed your screen and said you can’t enter any more chemotherapy until you put in this performance status data point, they’re just going to put in anything to keep advancing the screen unless it meaningful to their clinical practice – Or it’s audited – Okay, but I agree with that However, let’s think about it That’s if you don’t change the way people practice For example, if you sent out a survey to get information from people If that could start assembling the record If your nurse navigators or whatever,

instead of writing notes and throwing them away or putting them somewhere, actually the whole clinic, it’s going to require a change I didn’t say everyone doing the same old thing If you can do this in way that then when you come in, if you think about the the 30 people that are employed to touch a patient and do all these things, if you actually change their job and said, the primary thing is getting these data pieces correct and that would be how, that we would be paid and fueled, and you start using from the beginning to the end, then you could change I didn’t say it was going to be easy, and I didn’t say, but I think it’s possible But you can’t ask people to keep doing everything they’re doing plus fill out all this stuff And you can’t do it at the back end It has to be in the process of care It has to be instead of and you have to get something back for it – So I submit to you, think about this way, Laura We’ve got a 10 to 20-year vision, where I am in 100% agreement We have got to, really, as you said, re-engineer the whole thing But we’ve got a 21st century care mandate that says we’ve got to get these things figured in the next two to three years Let’s figure out, in the next two to three years, how to get our data to the point where we can get it ready for a series of responsibilities that we have in front of us And then let’s set the vision of how we’re going to start to meet these things, so that we don’t have to continuously hammer I think that was the word (laughs) – Even when you think about biomarker data or whatever, you think about it as raw process and signature data And you could almost think about, you at PatientsLikeMe are collecting a bunch of information that you’re trying to gain insights and eventually it will distill into, oh, this becomes something that ought to be routinely captured And it becomes a signature These are all, there can be many sources But you distill out not every piece of data Again, you’re not looking for perfection You gotta like really focus on, again, each person collecting those maybe five to 10 elements, period That’s it – That’s all I can do – That’s all they’re going to, but that’s okay You can do that because you don’t ask any one person to do everything – I can see a pathway where, I guess the business case, you might call it, is changing As Kevin mentioned payments changed where actually getting better results for patients, or tracking how they’re doing longitudinally matters The potential business cases around regulatory uses of real-world evidence, as you mentioned, Laura, would be a potentially strong motivator If there was clarity about what kind of data elements needed to be collected And maybe that would help overcome the linkage issues, too But there’s some other reasons why we’re not seeing these data come together, aside from linkages and consistent sharing, reliable sharing, being technically difficult and at least burdensome in the current system You all, Laura mentioned governance, how do you see that the different parties that need to come together for this, working more effectively together? Payers, patient data sources? Health system data sources? All seem to be key contributors to these critical checklist data elements that would be the foundation for reliable real-world evidence studies? Kevin, do you want to start? – Yeah, so governance is what I spend 80%, I think, of my day on In trying to figure out how to conduct data linkage Let’s be honest, the health system already knows how to effectively communicate real-world patient identifiable data to the insurance company Because UCSF or Duke likes to get paid And Anthem and other insurers like to know who they’re paying, so there’s already a handshake that’s out there that’s really, really valid, it has to be It amazes me looking at the back end of claims now, having been a pharmacist at CVS going, these things are getting rejected I’m amazed at how things get into the data sometimes But there’s already the pipes in place to do really robust linkage I think it takes, from a societal perspective then, being able to repurpose those already robust linkages to then provide for a framework to not only improve patient care, but then also build on the back end You talk about the five data elements that I might be able to tease a provider with If one of those is, did they get their flu vaccine? That’s currently wasting a lot of time in medicine Because if I just got my flu vaccine at my workplace insurance, that should automatically turn off the alert at the UNC or Duke system – Yeah, that’s right – But those little micro seconds of delay are killing us in time with the patient because that data’s not integrated Not the least of which, it’d be great to have real time vaccine exposure data, of course From a governance perspective, we absolutely have to come to the table and figure out how we can recontract, reunderstand that the needs need to be there for both the patients,

but then also for the research for the evidence to be built for a broader group – I’d like to just touch on a couple of things here I think one of the things related to governance is that currently, whether it’s in PCORnet or some other areas where we’re actually gathering some patient data from patients and thinking about how that works, a lot of it’s being framed within the common data elements that we’re familiar with with Sentinel and other databases that becomes part of There really is not a repository right now of patient-generated data within which other kinds of other information could start to be aggregated and collected So I think that that’s actually a real opportunity for us to begin thinking about One of the things that I think, and the way Amy describes it, is this value of the narrative and the value of the data coming together to tell a complete story But if we are only depending on common data elements, we will not get a complete story We have to find a way of being able to pull in these other pieces of information and ultimately, really nuance and add richness to that context In ways that other sources of data cannot fill that gap That has to come from the person themselves or the caregiver themselves to better understand what is that real experience when you’re not in the view of the clinical encounter? When you’re not in the system? When you are actually at home or in the place where you reside trying to manage in your day-to-day life? How can we help people gather kind of information and have that feed into a repository of data that ultimately goes beyond this common data model of standardized data, which I think actually ends up losing the story – I think Laura and Amy may have some comments on that – Yeah, so I’ll give you an example from our WISDOM study This is Women Informed to Screen Depending on Measures of Risk And the interesting thing is we have built in patient-reported outcomes and that’s the easiest thing to get Because we’ve done doing it going forward The hardest thing, I mean, the one data element that we we need which is res density, and if we are linked into the hospital, we can get it And for people who are coming from outside hospitals, that one piece of data is killing us And it’s not even like it’s hard to get and we’re going to do a natural language pass, and can get it in, but the whole process is about how you link people everywhere We’re going to get it from payers and then we would get it a year late And you can’t get it in real time Again, there’s this lack of ability to just have, okay, there are a few basic data elements that you really need to have And because you don’t have them, you don’t have them organized, and there’s no standard or requirement to have them exchanged or transferred It’s really hard, all this stuff is really hard All of us would say that It’s amazing we’re still standing, but it is – We persist – And that’s action, that the industry should crumble because of that They should know we’re out here and we’re going to solve these problems, anyway But I think that again, if you just had some easy data, ideas for data exchange that people required to do that, if you use IHE and force a data exchange and say, okay, the reason why I say it’s so critical on certain data elements that should be easily exchangeable, no matter what system you’re in, it would really help – I agree for the core need for a parsimonious data model and I also have seen, in spades, the importance of being able to add in the extra level of context and detail that can’t be gleamed from a common data model But I want to reflect on the governance question for just a second When I was at Duke and I’m looking right now at Adrian, governance was like 90% of our conversation Because, frankly, Duke could not do UNC – [Mark] Kevin said 80% and I heard Adrian laugh – My patient needed to nine miles down the road, to be in a clinical trial, and I’d always say it’s easier to get in the car with your medical record and drive it down than I can possibly figure out how to exchange The problem was really because of alignment of incentives around governance Now that I find we’re in a very interesting perspective So we aggregate data across our network, both on our electronic health record as well as electronic health records outside of our network But then clean it up and give back to the providers Frankly, we have almost no conversation around governance We all agree that the data really belongs to the patient and the providers, that Flatiron’s responsible for getting it cleaned up then we license all the extra clean data elements back to the sites But at the end of the day, because of the alignment of incentives, I’m a site that wants to do quality measures and understand data and be able to measure things within the context of my practice, there’s already incentive that has me within the governance model, in a way, that I didn’t have to worry about when really it was Duke and UNC fighting it out across state lines Or really, in state lines – In state – So the other part of this is that I’ve watched it get progressively more complex as we started linking in data sets So when, for example, we link in genomics data

and it’s highly aligned, because we’re now thinking together how we’re going to do research together that the organizations whose genomics data are being linked to our electronic health record data, we actually have no governance issues But when we start doing linkages where there are people at business odds with each other, the linkages get really tough So it’s actually putting on the table, what are the business issues? – Absolutely – And talking through it, because often you can come to a business solution, but you have to be really honest about what that’s all about before you get there So if I’ve learned anything, making this jump into the tech industry, it’s been how to actually manage the governance issue – Yeah, this does seem like an area where there is some more potential for convergence between these real-world evidence data needs and the shift that’s taking place in payment towards more accountability for results and the shift that’s taking place in patient expectations towards more ability to see quality of care and outcomes that really matter to them, too And again, it seems like the regulatory uses of real-world evidence could help add on an analogous or an additional source of momentum for these changes But I don’t want to leave this issue of governance and increasingly rich data linkages without talking about privacy concerns and patient trust concerns as well So somebody is holding this increasingly rich set of increasingly accurate data, which is potentially good for improving patient care, potentially good for giving patients insights about what they should do next, how they should make decisions Who is holding that and how do we pay due respect to patient concerns about trust and use of that data? Especially as we extend from patient care, which is governed by one set of rules to research which is governed by something else? – In this week’s news alone, on any time you have these major breaches of like Equifax, you know, you lose my banking, that’s okay It really sucks, but I can rebuild my financial But you lose the fact that I’m HIV positive or something, I can’t get that back That’s out there and that’s permanent So absolutely, patient privacy is at the forefront of every one of these discussions on governance Because it is this most sacred information that yet, patients all the time are joining Facebook groups and chatting about, but that’s their decision to share that information, they own that decision If we’re sharing data without the context of protecting that patient privacy, and it is lost or used in a way or manner that was not governed under the governance that needs to be put into play, you don’t get that back, you don’t get that trust back You lose that member, that member complains to their employer, you lose that business, frankly So patient privacy is of the absolute utmost importance in all that we do when we have the governance conversation – I want to be cautious, though There is absolutely no way we can assure people 100% protection of their privacy in the current environment that we live I think we need to be honest and transparent about that and remain open to the ways that we can ensure that the people who are the stewards of that data are held to really high standards That we have very intolerable thresholds of breaches of anything that would suggest that you did not do the absolute best you could to keep my data as safe and as protected as possible With the understanding that you know what, stuff happens And there are going to be times when that kind of absolute protection is not possible I think the public knows that I think patients, especially those living with chronic illness and especially those living with really acute and serious illness, are willing to, they have a benefit-risk ratio that’s very different than those of us who are healthy and thinking about how we protect our information in different ways For their perspective, care and research are intermingled – Exactly – They don’t think about this distinction between, oh, research means you have to generalize it, oh, care means it’s for me No, from their perspective, we should be doing all of that together in a way that ultimately gets us to the outcomes that improve my health – [Mark] That was Laura’s point, too – That is my point exactly, because in fact, I don’t think you should be allowed to practice without an eye towards improvement And I think you have shown at PatientsLikeMe, that people are absolutely willing to share their information, with some risk of loss of privacy – Big responsibility on our part – Especially at the beginning, to try and get people to pay attention to things that they were important So I do think one other thing that is actually interesting is that a lot of specialty societies have now been emerging as the holders of the data

There’s this huge ophthalmology database There’s one in cardiology, – I sense some skepticism coming up (laughs) – Well, I’m just saying, Cancer Link is now trying to organize all of the cancer data I have to ask, is that really the best place? Now I realize it’s not the era of big government, but I have to say I think this is a role that the government should play Whether it’s the FDA or it’s the, I mean, you answer this question, Mark I think that there has– – I’m just the moderator I get to ask the questions (audience laughing) – I’m asking it back to you But it’s like there has to be a place where we can sort of, where there’s some standards for who’s going to hold it The problem is that everyone’s got these hidden incentives to do things, there has to be a safe, trusted place for data to stay – Let me turn this one to Amy, since I’m the moderator But it sounds like – Oh no – two things that are coming out of this discussion, one is that patients are willing to share, but they need to have some confidence, that whatever system is helping to bring this together and support their care and support research is doing it responsibly and effectively and as safely as possible given that nothing’s going to perfect And second, some way of maybe actively saying, yeah, I want to participate in this study, that’s not maybe the traditional way we’ve done that kind of informed consent And I don’t know, Amy, if you were going to comment on that, but this is my effort to turns questions back to the panelists (laughs) – A couple of comments The first is, I think patients do often have a genuine interest in sharing, but they also have an expectation that the system has their best interests at heart If you read the Equifax articles, I don’t know how many of us sucked in our breath as we got to the place in the article where it says, and some of the senior execs were selling stock before the event happened I think all up until then, we were like, oh, gosh, you know, this is sort of like a foreign government breach kind of thing And then all of the sudden we got to that point and we we’re like (snarls) And I think that patients expect of us to absolutely be coming at this from the point of view of we are in it together to protect The second thing, and it goes back to your point, Laura, I don’t care if it’s the government or medical societies or anything else, whoever’s doing it, there’s the expectation there’s a technical solution to keep data safe The ophthalmologists, it’s not the ophthalmologists who are building the vault for the data It’s actually really a technical company that’s working with them So we need to be drawing on the best of technology to solve this problem, including new innovations for the future There’s a meeting going on not long from now, asking can blockchain help? We need to be really thinking about all the different ways to solve for creating the vault The third part of this is something that hasn’t come up, which is within the context of our current privacy legislation, HIPAA, and how that may get updated in the future, I’m not going to go into, but in the context of that, it is really hard to de-identify data because if you essentially just take out the named identifiers, it doesn’t do the trick If you’re going to do statistical or third party review, the truth is there are not enough third parties statistical reviewers in the country anymore to actually be able to verify all the data sets that are being generated So we actually don’t have a really good system, even just to manage the requirements of HIPAA in a way that is keeping step with the data sets that are starting to come out So we need to make sure that we’re driving all these pieces forward in a way that basically goes back to your core point Getting to the place where we’re doing what patients expect of us I imagine a day where it’s just like, on my Facebook, I’m able to designate what I believe I want to have happen with my information If somebody manages it poorly, I can rescind that responsibility of the right and in an electronic format, but we’re not quite there yet – I think we have to be a little safer than Facebook – Yeah, well, maybe, (laughs) but you kind of get my point, that the toggles are clear to me – It’s interesting, for our partner for the WISDOM study has been Salesforce and one of the reasons we picked it is because they actually have such fabulous security And the idea is we’ve tried to create this as a registry that payers have access to, patients have access to, the providers have access to That it would serves as an ongoing registry But if we’re really successful, and again, billions of dollars are spent on screening tests One, alone, it’s like a $10 billion a year and aggregate business If this becomes a national registry, where should that sit? Should that be at the CDC?

Could that be the FDA? Should be under PCORI? But it should be somewhere If it’s successful and grows into something that becomes this ongoing learning registry, then it’s gotta transition from a trial to something that’s what? – It still needs the right technical framework – [Mark] Technical framework, trust, effectively – We’ve got a good technical framework, we have it, but again, it’s going to grow up into these things – There’s nothing for it to sit within – Eventually, these things, if we’re successful at all, where does this evolve to? Where is the framework for this? – I get the sense I’m getting asked a question again here And I think I’ll actually take this panel’s discussion as being a success in laying out this important challenge and what the characteristics of the solution look like So we may not, in this panel, during this hour today, get to exactly what these long-term solutions are, but I think you guys have done a great job of teeing up not only issues, but the path forward And what the properties are that a solution needs to look like Which should help advance real-world evidence Now we do have some time, we could go on for a while, but we do have some time for questions and comments from those of you who are here A microphone is going around Marc, do you want to start? And then I see a couple of other hands up here, too? So, right here, first With Steve Steve, if you could just announce, just tell people who you are – Hi, sure, Paul Bleicher from OptumLabs And first of all, I have to start with a disclaimer, because otherwise, you’re going to think that this question is a little out there The disclaimer is that I come from the world of regulated clinical trials and archiving of data at Humedica and running very clean, precise studies at OptumLabs So I get everything But I’m playing off what I heard a little bit of today And it feels like we’re at Yahoo versus Google If you remember the old, old days, and we know who’s the winner there Yahoo had this carefully archived and constructed categorization of the web and Google came in and said, “You don’t have to do all that, just search it.” And find what you need and have a good search algorithm So we are in the middle of a revolution in data in the analysis of data And in healthcare data, specifically, around the use of artificial intelligence, specifically deep learning Which is fabulous at natural language processing and doesn’t require you to clean up the data or to categorize or standardize it or whatever Should we be skating to where the puck is going? Or should we wait and be like the healthcare industry and pharmaceutical industry have been traditionally, which is, let’s wait 20 years and we’ll adopt this Should we be thinking about a different way of analyzing data than classic frequentist statistics, but more make use of this very new, but very powerful way, and other ways of looking at data that can potentially give us better answers, not require us to spend all our time getting our data into perfect alignment? And be able to operate more efficiently and effectively? – Paul, thanks, and Amy, you want to do a short answer? And we’re going to take this, again, the purpose of today is not necessarily to solve all these problems, but to get the challenges and a potential path forward laid out clearly – So I think this is a critical question It is where the puck is going, in a lot of computing However, within health, we have the responsibility of getting it right and having systems that we know can get it right If we think about artificial intelligence, we’ve got basically supervised and unsupervised learning and the risks of unsupervised learning just really don’t necessarily hold up right now But if we can create data sets that are appropriately labeled that we can use to train our algorithms, and then build algorithms that start to essentially nibble away at the edges of these problems, we can systematically get there But it cannot be blacks not clearing Healthcare requires of us to show that we can get it right – Great, thank you, and great topic for the future I got Joe and I think Rich had a comment, too – [Joe] Hi, Joe Selby from PCORI Outstanding panel, lively and right to the point Kevin, I think it was you that used the word societal, that we need a societal change And Sally, it was you who talked about patients and their true inclinations and feelings and concerns

and what it would take And Laura and Amy, you talked about, I’m not going to remember exactly, I had it, it was all so nicely set up But the question is, you all alluded to the notion that we need a big societal change And I have come to exactly the same conclusion That whether it’s IRBs or whether it’s institutions talking to each other Would anybody care to just name some of the steps along that societal change pathway? – I’d be happy to address that I think the idea that you have to go to an IRB to look at your data to learn and improve is offensive to me That should be part of what you do as being a physician or a clinician I think the notion of the idea that we should have a system for research and we should have a system for care, I think is also outdated So I think that we have to start saying that, we have to start getting rid of, both in our public speaking to patients, oh, research is something you do over there The reason why nobody participates in trials is because we’ve got this big divide and because we have imposed this ridiculous, onerous structure on how we run trials Which is with huge overhead If we instead, okay, I hope next year I’ll be able to do things better than I do it this year So because of that, we’re going to ask you some questions We’ll make sure that you’re data is available to you and you can get back to it and we’re going to provide opportunities for you be in registries, for you to be in trials and we’re going to make it simple We have to start rolling some of these ideas out People have to start embracing ways of saying there’s a new way of practicing We have to start training physicians differently We have to give them different kinds of, a different mindset, as I said, mindsets, a different set of skillsets and a different set of tool sets so that people can do these things And it’s not going to be perfect, but we can’t just keep, it’s important to clean things on the back end, but it is important to change the frame One of the reasons why I’m trying to do this at the University of California is because they have six medical schools And it’s one of the ways in which you can start training the next generation of physicians to think differently and do things differently But I think it’s also all of us, in the way we speak about this The way you speak about what data is from patients and why it’s important to get data from them, but also back to them One of the things we’ve learned from patients is they’re willing to give us data if we give them something back They want the information back themselves So more of a shared experience And that people should be expecting to learn and look at different options and be able to have that information themselves – Just as a– – Can I just? One thing I’d like to add is that in fact, it’s not just a societal change, it’s a social contract – Yes, correct – That we do need to realign around and from what we’ve learned from patients, it’s their interest in understanding how they think about their health is how my mind and body are doing And help me measure that and help me understand what that looks like, whatever the measure might be And then also, we’re working really hard on this notion of thriving And when we’ve asked patients what does it mean for you to thrive in the presence of an illness? What they’ve told us is it means I can live the life I want So if we can help people better understand how to maintain the health that they have, even if it’s in the presence of illness, you can still have health But then put that into some framework that allows them to put that in context to live the life they want – And what can they do to really get better? – That’s realigning the social contract around each of us, frankly, not just patients, but just each one of us in this room – Yeah, that is a model of continuous improvement and how we support patient, continuous improvement in knowledge and think it will be very interesting to see as this real-world evidence work moves forward, how we make the incremental steps We’re not going to get there overnight, – It’s required part of practice, patient data, – But what’s the path forward? – clinician data, integration of data – I think we have, let’s see, was it Rich? Rich and okay, go ahead, Rich And then Marc and then Rich, you may not, I may have to cut you off – [Rich] I have my ways Hi, Richard Platt, I lead the Sentinel Initiative And I agree with all of the above But I would say 95% is the right figure for how much is governance and how much is technical (group laughing) – It’s getting bigger – [Rich] One of the key contributors to Sentinel’s success has been that we basically side-stepped that major issue of what data will you provide and how will we secure it? Because Sentinel is founded on the notion that FDA asks questions of the data holders and they answer the questions But they almost never provide individual level data If they do, it’s in a very circumscribed way I have been astonished over the past 10 years

at the progress that our methodologists have made in being able to do increasingly sophisticated analyses without actually having their hands on the data Or putting the data into a single pool data set These distributed analyses are getting closer and closer to the kinds of analyses you do with pooled data We now have a working prototype of a distributed regression model that should allow us to do any regression without putting data into a single place At the moment, it works across Sentinel data partners Can we do this kind of thing across Anthem and Humana and Etna? Both the theory and the prototype also work for distributed analysis where it’s linking the Anthem claims data to the PEDSnet EHR data without any of those sites actually releasing their grip on the data And it seems to me that I’m in favor of everything you suggested, but we also ought to look for ways to be able to work with the data where it lies – Yeah, in distributed models I’m going to leave that as an excellent comment to again help drive this forward And maybe Marc and Rich, if you both make your comments or questions and then we’ll see if we have time for a response – Marc Berger, this has been a fascinating conversation, but it seems to be that it’s been about, and I’m an IOM fan, this is about a learning healthcare system This is not about regulatory decision-making In a learning healthcare system, you can do what every big tech firm does You can do quick AB testing Or you can do control charts to see that if you make a decision, are you getting the outcome that you want to be getting? And why you get the right answer, it doesn’t matter as long as patients are getting better The right patient getting the right treatment at the right time The FDA doesn’t look at it that way They make a decision and they have to, they don’t want to make a wrong decision So if they were to give credence to some data, but it turns out to be not reliable enough, then we’re going to have regret and they’re going to get blamed for it So they don’t have the ability to go back and forth and make it go, so it’s a clash of cultures – But I don’t– – Let me finish – Yeah, we’re going to finish and this again, this is not going to be the last word on these issues We were just hoping to get the discussion moving forward – I think it has to do with the fact that we think of a regulatory approval as a big cliff that happens in the life of a treatment that’s being developed We have to move beyond that And the system is going to move beyond it Because the learning healthcare system, I know everybody wants to wait until we have the perfectly curated data, and like, it doesn’t matter The data is good enough today, it’s being used by everybody, a lot of people, a lot of institutions, today And they will change the practice of medicine much more rapidly The question is, can the FDA have some say where it looks I can now let people know about how the practice of medicine has changed? And I can let people know about it in the label? That’s the question? – And that gets back to our topic And I think actually you and Laura are largely in agreement on this notion of continuous improvement via data and evidence – It enables everyone to have a different role – So I want a last word question, Rich, over to you – [Rich] Well, maybe the advantage of going last is that everybody’s already touched on the questions I had Particularly the last two sets of questions Who knew this is complex, right? (panelists laughing) (audience laughing) And I think we heard that in a very excellent discussion And again, my congratulations also to the panelists We’re under a mandate and we have outlined the pathways And I would ask the panel to maybe close with what’s the timeline on such a pathway? And what are the quick wins that will allow us to really move forward in a meaningful way in a shorter timeline? – Yeah, let me ask that as a great closing comment And I ask you each to keep to 30 seconds or less, to keep us sort of on schedule But a quick next step to get us moving down this timeline? – Governance and getting the legal pieces in place to make these linkages happen It’s what I spend 80% of my time on I try to spend some time with my kids (laughs) – So while I agree with Laura in the long-term vision, we can get data sets ready right now, including with the parameters that should

be able to meet the expectations of OSI So three quick wins, we should be able to use these data sets for a label expansion or a label revision And what that happens in the next 18 months, I think that will be a huge change The second is when we put these data sets and serve them up as a part of a pragmatic trial that has regulatory intent And that will be another substantial piece The third is when we’re able to use these data sets to study populations and we’ve actually already got a really good example of this, who otherwise could not have been studied in the clinical trials For example, patients with heart failure who are in need of a breast cancer drug with potential heart risks So I think those are my examples – Great – Yeah, and I would follow up on the label expansion I think we can start working with that as a real potential target using patient-generated data Especially that data that’s being collected in a systematic, methodological, codified way as a early use case, to determine whether or not the data we’re gathering has insights enough to then look at other data sources that, as a compliment, or some other way of creating evidence from a host of different sources together But not including patient-generated data, not trusting that it actually does have value in a place within this evaluation, would be a real big miss And I think it’s an opportunity not to miss And I think we can do that in the next 18 months without working together – I think certainly once you make changes to the systems, everyone can start to have their roles evolve And for the long term, I think the FDA could behave very differently if they had different systems they could rely on But in the short term, I think the platform trials are a fabulous opportunity to say we can allow a therapy to evolve and we can think about accelerated approval with regulatory evidence generation And you could start taking, allowing trials that have had really clean data collection with early endpoints, evolve into dropping standards of care that actually perform in an inferior way and allow people to continue to go on where then you’re just starting to continue to gather the evidence You have your early endpoints, and you follow people to the end I think that is an easy early win that would be a big game changer And make people say, oh, I can change the way clinical trials are run, period and get faster – All right, I want to thank the panel A long-term vision on data (crowd clapping) Practical stuff to get there Thank you all very much All right, we’re going to take a 10 minute break, try to reconvene at about 11:15, thank you all very much – [Woman] Hi, how are you? (people chatting distantly) – [Mark] We’re going to start again in about two minutes

Two minutes, so those of you who are

out in the hallway, please start heading back (people chatting distantly) All right, we’re going to start up again in just a minute I’d like to ask everyone to take their seats Just as a little bit of logistics, administrative information, while people are headed back This session is going to run for about an hour And then we’re going to break for lunch Lunch is on your own, for those of you who don’t know this area, around 17th and F, there are lots of restaurants in the area and we hope to see you back in a hour after we do our lunch break Okay, so for this session, we do need to get started to stay on schedule I know there’s a lot more that we need to discuss today For this session, we are focusing on matching real-world data and real-world evidence to regulatory uses cases And I think a big part of this discussion is going to focus on specific types of real-world data As we focused on in the last session, but connecting to study designs, to analytic methods For those of you who read the white paper, you saw there was just as much emphasis on methods being fit-for-purpose as the data themselves being fit-for-purpose And that’s going to be the focus of our session right now

So how can we match methods, study designs and tools for analyzing data to the real-world evidence for use in specific regulatory decisions So with that framing, I’d to introduce our panelists David Thompson is Senior Vice President for Real-World and Late Phase Research, at IMC Researcg, InVentiv Health Marc Berger, next to him is the Co-Chair of the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Healthcare Decision-Making Which has done its own reports on related real-world evidence topics Jeff Curtis, is the William Koopman Endowed Professor in Rheumatology and Immunology and the Director of the University of Alabama-Birmingham Arthritis Clinical Intervention Program at UAB And Adrian Hernandez, my colleague at Duke is Professor of Medicine and Vice Dean for Clinical Research at the Duke Clinical Research Institute As in the last panel, we’re going to start out with some opening framing comments and a few slides from our panelists and then we’re going to go into discussion and again, I want you all both in the room and online, if you email us to get questions into the discussion that’s going to follow With that, I’d like to turn this over to David – Well, thank you, Mark Oh, very good, and thank you all for attending and it’s my pleasure to be a part of this discussion Very important event we’re having here So as you could see from my slide template, I’m a representative of the Real-World Evidence Project on the Clinical Trials Transformation Initiative And so I did want to open with a few words on that It was alluded to earlier today, but we didn’t really go into it too much But as the name implies, the idea is that we’re trying to transform the way in which clinical trials are conducted So the CTTI framework, it’s multi-stakeholder, it’s evidence based and it is designed to have measurable impacts on the way in which clinical trials are performed In was founded about 10 years ago, co-founded by Duke and FDA and has about 80 members Now the Real-World Evidence Project team is listed here, the CTTI Project Manager, Garret, is actually in the audience I put his face here without his permission (audience laughing) So that if you want to find out more about what we’re doing with the Real-World Evidence Project, you can hunt him down later and he has assured me that he did not exhaust all the names at this list before he finally got to the bottom and chose me to represent the entire group In any event, so far what we’ve been doing is focusing on some of the definitional issues with respect RWD and RWE And a set of interviews are being implemented across various stakeholders to try to better understand their perspectives on data needs, evidentiary requirements Specifically to help decision-making and of course to incorporate this into the way in which clinical trials are conducted And there’s been some discussion earlier today already about the inefficiencies and the high cost of clinical trials With the data that we have available now and the technologies that are available, there’s got to be some ways in which we can bring these together to develop some greater efficiencies So I’ve been asked specifically to comment on use cases for real-world data in early development And the interesting thing about it is I’ve been doing retrospective database work using real-world data sources for nearly 30 years now and for the most part, it’s always concentrated on products already on the market Phase IV, because by definition, you can’t really collect real-world data on a product until it is actually on the market and being used But there are instances and we’ve some references to them today about the way in which real-world data can be used for products early in development Specifically from the regulatory standpoint, we’ve already talked about how on an occasional basis, FDA will accept historical controls from real-world data sources And this tends to be in instances in which it’s either infeasible and/or unethical to create a control group for a particular trial Rare diseases is the most common example used there

Beyond that, my work has focused on helping manufacturers understand the competitive landscape in which they are going to enter as relates to using real-world data to understand things like treatment patterns and drug adherence for products currently on the market that they may be competitors with as their clinical development progresses And so the questions that arise relate to, is there effectiveness, efficacy-effectiveness gap? Are there differences between how products perform in clinical trials versus how they perform in actual clinical practice And that’s a phenomenon that has to be taken seriously We need to measure that, we need to assess that And it can be used to inform the next wave of clinical trials for competing products that are coming down the pipeline Also, with the size and vast volumes of data and the greater numbers of patients that we’re seeing receive products, once they’re out on the market, there’s greater potential for doing patient segmentation An assessment of heterogeneity of treatment effects So we might find that there are particular patient populations that aren’t benefiting from a product Even though they might have been represented in the trial population, but are being washed out by the average treatment effects being reported in the clinical trials We have the capacity, we have enough cell size, across a range of patient strata In order to be able to look at differential benefits and harms of different drugs when used in clinical practice And we use that as a basis for assessing whether or not there is additional unmet medical need In addition, there’s been a lot of discussion and it seems like a low-hanging fruit area would be, for 21st Century Cures, is to potentially use real-world data for label extensions So what we’ve done historically, though, essentially in the category of hypothesis generating activities There’s a whole body of comparative effectiveness research in which we’re able to compare products using quasi-experimental design techniques And some of the other statistical advances that have come down the pike to address issues of confounding and bias And we’re able to do product A versus product B comparisons and assess treatment outcomes and costs Historically, that’s been specifically for hypothesis generation activities that could be confirmed in later clinical trials But potentially, in the future, this is an area where FDA might actually evaluate this kind of research for decision-making purposes And then finally, I come from a CRO and we frequently use real-world data sources to assess protocol feasibility Specifically, this relates to when a protocol comes down the pike and we have inclusion/exclusion criteria We can take those criteria and overlay them onto an existing database to see how stringent certain of the criteria are, in terms of filtering away the available patient population And we could use that to tweak the potential criteria for selection of patients so that we have a feasible protocol to implement in the field Now when we start talking about where things are going in the future, there’s some reference to this in the white paper that’s been cited a couple times today And one of the things we’re talking about within the City Transformation Initiative for clinical trials is to what extent, not only the data, but the IT systems that house the data To what extent can these be used to transform the way in which we are conducting clinical trials? So we can use these data systems to identify patients who might be candidates for inclusion in studies And this really shifts the trial paradigm on its head Because historically, we identify sites and investigators first and they enroll patients But now we have the capacity, using the technology, using the data, using our analytic methods, to look for the patients first and then reach out to the physicians We have also, through those electronic medical records systems the communication channels in place to then reach out to those providers And the providers are the only ones who the patients are because all of our analyses are done on encrypted data So the patient identifiers aren’t available to us, but they are available to the providers So it’s a two-step process for that outreach And then finally, once the data, once the study is under way, we establish

some connectivity between the EMRs themselves and the electronic case report forms to try to automate some of the data capture and reduce redundancies in data entry I have a little graphic that summarizes that and I’ll just demonstrate that once you have an EMR system, the communication channels can go in the opposite direction You liked that, huh, Mark? I always want to amuse Mark (Mark laughing) (audience laughing) And so you have the capacity to use the provider network that is in place as a consequence of them all being united in their use of the same electronic medical records system And then you use the communication channels that are established electronically between the EMR provider and those practices and then have then, collected the patients included, it’s everyone in the practice on the one hand and it’s selected based on protocol in another Provider-induced variability in data collection Big differences, practice-based customization of data collection It’s encouraged by EMR systems It’s highly discouraged in trials The project that is ongoing, so those are my opening comments and thanks very much – [Mark] Thanks very much, David And next is Marc – So I’m here today representing the ISPOR task forces that have reporting, we’ll have two papers out this month online And just to let you know what ISPOR is, in case you don’t know, it was founded in ’95, with a mission to promote health economics and outcomes research excellence to improve decision-making And it has to grown to become one of the leading global scientific and educational organizations And as we heard in the last talk, there are many different stakeholders And ISPOR thinks that all the groups around the circle need to part of the solution as we build a learning healthcare system All right, so the challenge of real-world evidence There is tons and tons and gigabytes and gigabytes of real-world data This is a picture of the background radiation in the universe We think it’s reliable, even though it wasn’t collected in a randomized controlled clinical trial But how do know when a data is reliable and trustworthy? That’s the central question for the FDA Is it good enough? Is it reliable enough that enables you to make decisions beyond what you’ve been able to make thus far on safety or for rare diseases? To make real-world evidence useful, it has to produced in a quality way, there has to be good data collection, good analytic methods, transparent study procedures to enable replication, good procedural practices, which I’ve called study hygiene And then responsible consumption, which means informed interpretation and fit-for-purpose application What I’m going to focus on here is the last two sub bullets under quality production Because I think there has been concerns regarding the replicability of observational studies And whether or not there isn’t some data drudging that goes on to cherry-pick good results So how do you know that hasn’t happened? As I mentioned, this month, coming out in joint publications in Value in Health and Pharmacoepidemiology and Drug Safety, there are two task force reports for joint task forces between ISPOR and the International Society of Pharmacoepidemiology, ISPE One about recommendations around good procedural practices to ensure that you think that there is greater trustworthiness around results that you get from a study And one focusing on the replicability, so that you can say, hey, if I had the same data set, would I get the same result? I’m not sure if it was a year ago or two years ago at one of these Duke-Margolis meetings, I put forward the fact that we had good procedural practices in place for RCTs and they were needed We wanted to make sure that pre-approval RCTs were pre-registered on a public website, We wanted to make sure there was completion of an a priori protocol and data analysis plan Transparent documentation for any changes in study procedures And expectation that all RCT results would be made public There were no well-accepted recommendations for good procedural practices for observational studies

or real-world data studies And we saw this as a gap that, it’s not the whole solution, but as one step, if they’re adopted, that would lead to greater confidence and trustworthiness given to these kinds of studies Now a few groups have been talking about this, about pre-registration, but it needs to address not just publication bias, because we know journals like to publish positive results, not negative results And it has to address data dredging Can you torture the data ’til it confesses and tells you something you want it to say? And then there are other concerns around internal validity, inaccurate recording of health events, opaque reporting We believed that following the RCT-like practices is a logical starting point Now we’re not trying to tell everyone that every observational data analysis that you do needs to follow these rules So we divided the universe of RWD studies into two different categories Exploratory studies and hypothesis-evaluating treatment effectiveness studies or HETE studies Listen, everybody don’t, when they start playing with data, they don’t always know what the strength of the data is, what the weaknesses of the data, what questions you can ask You explore the data and you generate hypotheses Nothing wrong with that But those are not the studies we’re talking about If, on the other hand, you want to an observational study that you think should inform decision-making by any healthcare provider, then you should be having a a priori hypothesis and a established protocol because you’re going to evaluate the presence or absence of a pre-specified treatment effect and/or its magnitude And if you did that, and you said you were going to do this in advance, then I think that you should have greater confidence that those results are not the results of random findings, but actually represent as good as we’re going to get And if you don’t rely upon a single study, but say, let’s now see that in multiple studies, you know what, that’s kind of what a learning healthcare system’s going to be So we’re going to focus on the HETE studies And here are the recommendations That HETE studies should be pre-registered A study protocol analysis plan should be on a public registration site prior to conducting study analysis The publishing of study results with attestation to conformance and/or deviation from the original analysis plan Everybody modifies their analysis plan to some extent, even in clinical trials, as they go through You should be very explicit about you did, why you did it and what you found There should be the opportunity to replicate findings Many of these data sets are commercially available So if you’re transparent enough about how you did the study, other people should be able to go in and get the same result If you do an exploratory analysis off of a data set, you shouldn’t use that data set to then go do a confirmatory analysis Why, because you may just be showing what you found in your exploratory data set is not generalizable beyond that data set So it should be done in a different population when feasible Now sometimes it’s not feasible If you do a study off of the Sentinel data set, you could be hard-pressed to find a data set as large and rich, Platt-ish, as that Hi, Rich (people chuckling) I think the authors should work to address methodologic criticisms This is an evolving area The methodology is rapidly evolving and there’s still questions about is the methodology always good enough with a particular analysis? I think a robust public discussion about what are the pros and cons, need to happen And then including key stakeholders, not the least patients, but caregivers, clinicians, administrators, HTA/payers, manufacturers It should be relevant information to everyone who wants to see a learning healthcare system The ISPE-led report focused on enhancing reporting guidelines to identify the minimum set of items necessary to report in detail to achieve a fully reproducible evidence from large healthcare database cohort studies I think we’ve been pushing in this direction for a while, but I think this report goes into much greater detail than has gone into before

I’m not going to go into great detail about this, but suffice it to say, there’s specific decisions around analytic data extraction from raw longitudinal data with a focus on temporal anchors The minimum reporting of independent investigators to be able to reproduce the data cohort study Starting from an analytic data extraction from a longitudinal raw data set And the reporting on the analytic cohort before and after adjustment There are many things in this report It builds on some of the work from RECORD and others and many good recommendations that have come out from various groups around the country and the world So closing thoughts, I think as a first step, enhancing the trustworthiness of real-world evidence requires us to have good operational procedures To make us believe that this wasn’t the result of data dredging and that people know explicitly what you did and actually, they can go out and reproduce what you did I think these are good basic steps It will not take place without a variety of stakeholders, including journal editors, regulatory authorities, providers, payers and HT authorities putting the right incentives in place But as we heard before, and I don’t think it’s only about money incentives Yes, money talks, nobody walks But there are many kinds of incentives that could be good The HITECH Act, look how it’s changed a number of EHRs that are being used in this country in such a short period of time With the right incentives, you could move people’s behavior We have an upcoming meeting that’s going to be on October 20th, here in Washington at the Grand Hyatt Hotel Which will be jointly hosted by ISPOR and ISPE In which we’re going to present in complete detail the findings of these two task force reports And begin to start a dialogue with various stakeholders about how we can get the enterprise of real-world data analysis to resemble more what has been done for clinical trials Thank you very much – [Mark] Thank you, Jeff? – Well, thank you very much for the invitation to come spend some time with you and I guess by way of disclosure, if not apology, I’m a rheumatologist So most of the use cases I’m going to describe are out of my field that have personal experience with But the principles are generalizable I was asked to take some of the concepts that we’ve speaking about throughout the morning and bring a few of them to life And sometimes when these topics come up in conversations, they seem very futuristic and it all sounds great, but who’s actually doing this and what are the regulatory implications of doing this type of work? Do you have real uses cases or is this still all in a theoretical framework? So I’ve taken four different domains as described on this outline to just bring you a few examples Pre-licensure or an expansion of an indication, pragmatic clinical trials and our two other topics So I study mainly rheumatoid arthritis or RA and psoriatic arthritis and conditions that are fairly uncommon that have a population prevalence of roughly 1% So if you want to study safety and comparative effectiveness in an uncommon disease with rare outcomes, you better have big data sources And you’re not going to get safety out of clinical trials Real-world data is particularly helpful to understand the safety profile, especially when there is very limited information on the background rates of some of the therapies that we use in medicine I’ve given two examples here, one of them was a new first-in-class biologic that targets the interleukin 6 receptor inhibitor We’d never used this molecule anywhere in medicine before The first use was in RA and some of the safety concerns related to liver issues or liver enzyme abnormalities and hyperlipidemia And post-marketing safety commitments were being discussed and real-world evidence, including claims data, was used to inform what a five-year safety steering committee was going to look at and the study design that would have regulatory implications Maybe even more on point, a different molecule that’s a targeted therapy that effects the Janus kinase pathway currently approved in RA was seeking a label expansion for a different disease, psoriatic arthritis And while the safety profile concerns were somewhat known, there wasn’t a lot of information about the background rates of those events in a different disease So real-world evidence, including health plan claims data, was presented as part of the FDA package that was submitted in the dossier for a label expansion That was presented a month ago to the FDA and the Arthritis Advisory Committee as part of the background to inform and contextualize the safety for the label expansion for a new disease

for a drug that was on the market, but approved for a different condition And I’ve given the references for that In terms of the conduct of trials, so we at UAB and about 20 to 30 other sites are working on a large pragmatic trial This is intended to evaluate the safety and effectiveness of the live Zoster vaccine among people age 50 and over on TNF therapy This, frankly, from a clinician’s perspective would be a no-fly zone because this is a live virus vaccine And the reason we don’t use this much in rheumatology for people on immunosuppressive or modulator drugs is the safety concern If you’re going to have something bad with a live virus vaccine in an immunosuppressed population, it’s going to happen pretty quickly Like in the first four to six weeks It’s a very simple trial, if you’re over age 50 and on one of five anti-TNF drugs, pretty much you’ll qualify And the safety concern is going to manifest in the first four to six weeks So that’s the pragmatic bit From the site’s perspective, each patient is done at six weeks But one key question that is of great importance is how long does this vaccine’s protection last? So we have follow up with a linkage to claims in EHR data for people as part of the consent form So that you don’t have to keep people in this study to figure out, hey, what’s the rate of shingles? What’s the rate of postherpetic neuralgia? And this pragmatic trial is under an IND from FDA with the understanding that it would result in a label change if the data that accumulates is as hypothesized This would be part of that regulatory submission and the FDA accepted this study design We’re using an electronic consent system to help with screening and make that efficient and to randomize people in real time The part that I wanted to just spend a minute on that this electronic consent includes an authorization both to obtain medical records centrally, sites don’t have to do that, and to link to external data sources that might include health plan claims and EHR data like that from PCORnet As part of the trial’s feasibility, we did basically what David described just a few minutes ago You have enough people in the country that might do this, so we used Medicare claims data and Medicare data, like everybody’s health plan data, knows who the doctors are and how they practice, because that’s how they bill So we found eligible patients on therapies of interest, grouped them into their treating physician, grouped the treating physicians into their practice, and then we said, well, which doctors have signed a 1572 form and do research and then group that by the largest to the smallest offices And that’s what this picture shows The X axis is the number of doctor’s offices that do clinical research that have signed a 1572 form that you would need to screen the Y axis, which is the number of eligible patients for the trial So depending upon what estimations that you make regarding how many people will say yes, 25, 33, 50%, et cetera, that tells you how many sites you need So we actually figured out both how many sites we need, but then because you know who the doctors are, we actually then can go to those sites and say, hey, do you want to be in this study? At the individual sites now, we are then using the same kind of approach to essentially pre-identify people Rather than the doctor referring the patient to the study coordinator, where he or she thinks about it one patient at a time, that doesn’t work very well The study coordinator can tell the doctor, “Hey, I’ve searched our data using tools like i2b2, “or SHRINE, I already know who the people eligible are.” And then the doctor is informed by the study coordinator, this patient will likely qualify So that really helps with screening Electronic consent standardizes what the trial is about There is a cute little six-minute cartoon It’s delivered the same way to every patient, not with the vagaries of what a study coordinator, or even what the PIs would tell the patient And then we have a knowledge review, which is really a quiz, but it sounds better as a knowledge review (audience laughing) Do you get it, as a patient? And if not, then we bring them back to the consent This is all on the tablet, this is not taking the study coordinator’s time, to remediate that because we really want to get it And there’s no paper data, there’s nothing to have to upload via a paper CR case report form into an electronic data system It’s all on an iPad tablet And that same tablet is used with its camera, so that if somebody manifests a rash that might be clinical shingles, that’s part of the study data that then gets sent to an adjudication committee to decide, hey, is this shingles or not? So electronic and digital images are part of the study repository As part of the consent, there’s a HIPAA authorization and electronic medical record release form So that’s where we, the central studies coordinating center, can go get the medical records We don’t bother sites to do that because we want to make it really easy for the sites

There are other examples and I have a reference at the bottom for a different study we were involved in This was a five-year head-to-head study of a new biologic versus a TNF therapy And it had a cardiovascular safety outcome Part of it, there was intended really to inform the safety, that’s really the bulk of why it was done But patients might not keep coming back and people switch therapies So they were asked to sign a consent that allowed the pharmaceutical company as well as the clinical research organization to have the identifiable information to go get the medical records And I remember when my study coordinator saw this She called me and said, “Dr. Curtis, we’re releasing “to a drug company patients personal information? “A, they’ll never agree and B, isn’t this an awful thing?” and I had to explain to her, look, you’re looking for a heart attack and if you have a 3% loss to followup each year, that’s probably going to exceed the event rate and we can’t have that That compromises the whole point of this five-year, 3,000 person plus study and that’s why it’s so important It takes a bit of extra time, but this is a nice example where the same approach is being used as part of the post-licensure safety commitment As part of this same notion of making it efficient to do safety follow up, Sentinel of course is very effective to find a number of adverse medical events There’s algorithms and claims in EHR data, we’re all familiar with these sorts of things But you can use the same approach not just to find outcomes using data claims or EHR by themselves, where maybe you want to maximize specificity, but rather just for case finding Have an algorithm that dials up the sensitivity, finds the universe of every possible adverse event out there that you’re looking for, but then because they signed this electronic medical record release form, at the baseline visit, now you can go get all the medical records You can centrally adjudicate them, as we normally would do And that’s how we’re working it in this pragmatic trial And nobody has to come back for safety visits year after year just to ask questions like, are you still alive? Have you had a heart attack? Have you developed cancer? Because you can find those things in real-world data And that helps minimize loss to follow-up because patients don’t have to keep coming back year after year for safety follow-up IRBs are okay with this, and we and probably many in the room have done a number of linkages with real-world data Example language if you’re going to collect Social Security numbers from one of the studies we did where linked claims data to a large registry, is shown as described You don’t have to have Social Security numbers, people are very skittish about that for reasons we all know Methods to use multiple non-unique identifiers, sex, date of birth, worked quite well You can use an honest broker, if you need to You can use hashing algorithms and fancy computer algorithms to do this and that works fine The last example in the last minute or so, is the post-marketing safety commitments That if you really need to study something rare, and this is an example from osteoporosis, where a first-in-class biologic drug, denosumab, when we use it for osteoporosis, it’s called Prolia One of the things that was of interest was to study osteonecrosis of the jaw The event rate for that is less than one out of 1,000 people Claims algorithms by themselves do not work particularly well So we at UAB contributed Medicare data, Optum contributed United Healthcare data and then there’s a European data source Those three data sources that have some targeted medical record retrieval in case adjudication, that forms the major bulk of the safety commitment that the agency accepted as part of studying the long-term safety of this first-in-class biologic therapy So this is perhaps among the better examples I’m aware of where this kind of data source, this is claims plus a little bit of targeted medical record review, helped inform some of the regulatory implications about the safety of this biologic product With that, I’ll turn things over to Adrian – Thanks, Jeff Okay, I’m actually going to summarize with really three stories here I think one of the themes that you’ve heard on this panel as well as earlier, is that the real-world evidence is about the totality of data It’s not a single study or a single trial or a single observational study Really what we’re talking about is creating a living textbook about a medical product and/or patients that have experience with that medical product Whether it’s in formal research or whether it’s in experiences in the real world And if you consider an example, and I’ll use my mother as an example, she has premature coronary disease She is very interested in improving the health of people like her with coronary artery disease However, clinical trials is not a convenience for her life She cares for four grandkids and does an excellent job for us

and she loves doing that every day Now can you imagine her coming in every week or every other week for a clinic visit where she has to park, where she has to find her way to a site, a study clinic, that’s usually in a back corner And then wait and wait and wait to donate blood to answer a series of questions That’s unrealistic and that’s what you hear all the time that people are highly interested in being part of research, but when it comes down to actual participation, it’s very much the opposite Now if you offer a clinical trial, and I’ll use ADAPTABLE that we’re doing with PCORnet, testing two doses of aspirin, where it’s actually at the convenience of a participant doing the trial Where someone is being electronically identified, electronically contacted, consented, randomized and follow up, you can have a very simple system where more people can engage and participate in research Now it is important to have the full context of what’s the benefits at risk, in the case of ADAPTABLE, we’re focused really on the benefits of potentially less MIs or less death or the risk of bleeding So it’s that benefit-risk ration that you really have to understand across the context of different people Now often, in real-world evidence, we’re talking only on the focus of the benefit Most of the time we actually need to understand that risk-benefit ratio But we don’t necessarily need to think about it where we’re as worried about the unknown unknowns So as we develop study designs with, whether it’s NIH or other sponsors, people are very interested in doing trials that are simple, pragmatic and we can easily define the types of endpoints that are patient centered that are convenient to collect, that are able to be collected in the healthcare systems, such as what Jeff was describing Or with claims, but where people have problems is starting to think about the so-called unknown unknowns Are there safety concerns that people have that so-called haven’t been discovered yet, even though there has already been a wealth of evidence that’s been developed around the medical product So in the case of aspirin, well, actually, there’s been over 100 years of evidence around the safety of that So we actually know what we need to focus on Now if you started thinking about that context of benefits and risk, then think about other case examples And I’ll use heart failure as another example Traditionally, and for good reasons, the focus of drug development has been to improve the longevity of life as well as to prevent a major complication such as worsening heart failure However, patients also care about how they do every day Can they walk more? Can they do more activities? Can they have a better quality of life? If you think about that as an example, where a medical product gets on the market for heart failure and having cardiovascular benefits, in terms of reduced death or hospitalizations But still, the interest from patients is, actually, can I walk longer? Can I do more? Can I spend time with my grandkids? That becomes harder to do if you start thinking about what would it take to do another randomized trial to extend the understanding within the context of heart failure Such as using devices that actually measure accelerometer data to actually learn about someone’s experience every day, every week, as opposed to a six-minute walk Which you’d have to come into clinic to do on most cases Or learn how their symptoms are doing every day and the trajectory of that In that context, you could easily do a pragmatic trial having things that are collected at the convenience of a patient, but then what gets people worried about is the so-called unknown unknowns in terms of, well, how would we collect this type of SAE and report it in this fashion, et cetera And so that’s one of the challenges Now think about another case, diabetes In 2009, there was guidance that was released that actually really tried to address the question that had come up as a concern where medical products for diabetes were focused on lowering hemoglobin A1c, but there had been concerns about cardiovascular safety There were a series of trials that were actually done in that context And then now, actually what we’re seeing is actually trials showing real benefits in terms of heart outcomes Now you start seeing that scenario and start thinking about, well, where do the benefits really extend to? Many of those trials were in narrow populations

But then there is populations that are related to patients with diabetes, so those, too, are at risk of diabetes Or another example is patients who have heart failure who could potentially benefit from a specific medical product that showed that benefit in trial Now to actually answer that question, if you were required to do the same size trial that got the medical product to market and cost hundreds of millions of dollars, that would be a real challenge here You actually have potentially a great profile in terms of the safety, in terms of the real world You also have a great profile in terms of what it does in its initial indication as well as cardiovascular safety So really trying to understand what’s the benefit to risk in patients who have related conditions Heart failure or chronic kidney disease are two examples there And then if you start thinking about pulling all this together, in terms of totality of data, Sentinel is a great example, where it’s constantly evaluating safety of medical products as it enters the real world So if you’re doing a focused clinical trial in an expanded population, understanding benefit and risk for a condition that’s related and you also have a parallel system that’s evaluating safety, putting that together as a total package for real-world evidence can really be very powerful So we are really having a learning healthcare system and creating a living textbook that people are making decisions Patients, clinicians, health systems and regulators – Great, thanks I want to thank all of you for, what I hope has come out from these presentations is a couple of things, at least One is the wide range of potential real-world evidence use cases or applications relevant to regulatory decision-making So we’ve seen a spectrum here in the examples and the concepts that different panelists discussed and presented, and I think the second thing is that this is all happening now, across this whole spectrum And I’m going to a followup question each of you in minute And it’s going to be about a practical next step that FDA and those working with FDA on real-world evidence can take to advance the kinds of use cases that you’ve described But I just want to frame, briefly, and connect this back to the framework that we described in the background paper, which are trying to build on With FDA and all of you There was a spectrum of uses described here, ranging from pre-market studies of new medical products We heard from CTTI, from VERVE, I think, to some extent, too, about this notion of going from sites to patients to patients to sites Sort of flipping the trial design to make it more about trials that can be conducted in real-world care delivery settings And that fits with what you all heard in the first panel about this notion of trying to embed research and learning into the healthcare system That requires this kind of flipping and that is happening now through the kinds of pragmatic clinical trials described here, Sean Tunis and others in this room who have worked on a range of others This is happening now With randomization, but done in a real-world setting That is real-world evidence We’ve also heard the other extreme of observational post-market study commitments to address safety questions, learn more the, addressing the unknown unknowns or getting a better idea and making the unknowns about the knowns more precisely understood Some of that can involve, and often does involve, non-randomized observational studies And what I think what Marc covered, that was kind of a spectrum between there I think a lot of the ISPOR-ISPE principles could be applied, and I think you had in mind thinking about how to do non-randomized real world studies effectively Maybe going beyond just post-market surveillance applications to other kinds of label extension or modification studies and add a set of principles and ideas, a set of principles and approaches for good study hygiene If the kinds of principles and ideas that are behind all of the applications that you’re hearing about here could be better understood, could kick the tires on, could be a clearer part of guidance on developing real-world evidence for regulatory uses You can imagine a potential path to transforming how we’re collecting evidence and with a lot of benefits in the process

So we seem to already be on this journey across this spectrum, what’s a next step? Based on each of your experiences that we could take to accelerate progress somewhere in this broad range of use cases? In this potentially broad range of going from between prospective and other good observational practices, all the way to some version of randomization? – Yeah, I’ll get us going I think that we have the technology We have the data We have the knowledge to bring it all together to inform decision-making I think what we need from the FDA side is some structure and some guidance on what kinds of information is going to be acceptable To me, the blending of traditional RCT data on the one hand and observational data on the other, in the middle there is the pragmatic clinical trial And when you think about the changes in the regulatory guidance in Europe, where they’ve invented a new concept called the low-intervention clinical trial When you look at that and some of the aspects of it, it’s kind of a euphemism for the pragmatic clinical trial It’s for products on the market used in terms of typical clinical practice And there are differential kinds of regulatory requirements associated with that Currently though, in the United States, we don’t have anything akin to that I think some guidance along those lines will enable us to take that step towards greater use of the pragmatic clinical trial design as an approach for real-world evidence generation and use for regulatory decision-making And I would argue that that’s going to be the kind of real-world data that will have the greatest level of rigor and comfort level for all concerned Because it still retains randomization, no matter what kinds of methods you throw at the observational data, you’re never going to get rid of all the uncertainties associated with confounding and bias But if you have randomization in place, miraculously, it seems to work So I would like to see us move towards greater use of the pragmatic clinical trial design in the future – Marc? – Well, you know, if we’re going to have a learning healthcare system, then it can’t just use the data that’s thrown off by the healthcare system just to assess safety And we’re very comfortable with that Pharmacoepidemiology has established good practices about how you do signal identification, signal verification, that sort of thing On the effectiveness side, though, there’s been a lot of concerns Some of which has to do with the fact that the effects sizes tends to be relatively small Having said that, this data will be used by many, many different stakeholders across the healthcare system And the question is, can we make it more credible to a wider variety of decision-makers? And I think that just as there were concerns 10 years or so, or whatever it was ago, regarding the randomized clinical trial engine and it led to the International Consortium of Medical Journal Editors to require pre-registration on if they wanted to be published And that the FDA, if you want to be considered for it, pre-register it I think that the community that’s using the same techniques that have been used for 30, 40, 50 years for health services research and pharmacoepidemiology, but are now applying it to understand what are the benefits? And who benefits the most? I think we can move substantially in that direction if the consumers of those, potential consumers of that information, be it journal editors or the FDA or payers or HT authorities, require pre-registration It’s not an absolute guarantee At the end of the day, if people want to do fraud, they can do fraud And fraud still happens today in all aspects of scientific endeavor John Ioannidis talks about this quite well But I think that most people are of good will and good spirit And I think that if you say that the standard, what it will be, that you know what, explore the data, but if you want to make recommendations,

that you think are worthy to change healthcare decision-makers ideas, then you should follow some good procedural practices And I think if we did that, over time, it will transform the entire culture of health services research and outcomes research – Jeff? – From my perspective, I think that the FDA has been open, but it would be exceedingly helpful for an even greater openness as well as an impetus, if not a mandate to go expand the kind of trial endpoints that are going to be much more informative to patients Your mother wants to spend time with the grandkids, how do you measure that? Could that be a trial endpoint? She doesn’t care what her ejection fraction is, I suspect, and event rates for heart failure hospitalization or mortality, that’s rather crude And that’s not really what she cares about, so I fear we are doing trials that don’t really matter to patients, by and large But when those kinds of outcomes, and assuming you can quantitate them through the NIH PROMIS system or other types of things like that, when that gets discussed or vetted, as the trial’s outcome, I often hear, well, the FDA doesn’t accept that They’ll accept the SF-36, and to my knowledge, at least in my circle of influence, there are exactly zero clinicians who routinely measure the SF-36 at routine care visits So you can get a label claim for patient-reported outcome improvement using a measure that nobody really even knows what it means and nobody uses it in the real world So to me, I think that’s really where we need to push the envelope We need the informatics systems to know where to stick that kind of data using controlled terminologies or ontologies, and not to have everybody doing their own thing We need an expansion of LOINC or something like that to know how to handle that kind of data And PCORnet has made some end roads in that regard But to me, that will help us do trials that actually matter to patients In a way that the data can be more interoperable – So I can see really three things One is predictable pathways So I think what we often hear is that there’s strong interest, the data’s there and in many situations, randomization is very important and could be done But what happens is concerns about unpredictability in terms of evaluating benefit to risk and so therefore, it causes a lot of extra measurement of information that’s hardly ever used As an example, as clinical study report may be this thick, the top line results, or it could be done in a pages at most, and that’s people make really their decisions So predictable pathways The second thing, and it’s tying off the CTTI recommendations is right now, there’s a lot of interest in how can you convert the real world to the FDA world So the example is electronic health records That doesn’t operate with CDISC Clinicians don’t operate in that way, either So how do we actually take advantage of what that is in terms of the real world as opposed to trying to get everything converted to something that is a standard that is not necessarily part of clinical practice And then the third thing comes out of actually taking advantage of the lifecycle of a medical product As you get further along, in terms of lifecycle of the product, the incremental information that would be gained through doing the same study again, say, for a safety component, it’s going to be really less I mean, it’s going to have a hard challenge in terms of saying we’re going to find something else Especially in a setting where we’re having other evaluations that are ongoing, such as with Sentinel – Great, thank you for the comments Now we have time for a couple of questions This was kind of a methods heavy session, given the range of uses and potential best practices and next steps We have time for a couple of questions and comments on these topics, Bob – Wait for the mic? – Yeah – Boy, am I surprised This has been very interesting Some of the kinds of uses people want to put are conceptually easier So I grant that the data you get from an electronic health record isn’t the same as what you get from a clinical study report But in a randomized trial, I’m convinced that a lot of those conceivably could be credible Survival, obviously would be, but so would some others Whether you had a heart attack or not, like that I think that that’s not so hard The hard part, I think, is when epidemiologic data is going to be acceptably reliable or credible without randomization, for an actual claim That’s the hard part

And I guess I haven’t heard so much about how to do that One thing for me, is at least do it twice in two different settings That was said before by Marc, I guess But also the other thing is to take some completed conventional trials that have been done and see if you can replicate the result in the real-world setting using epidemiologic findings Without knowing too much about how they did the other trial, it has to be sort of blinded And see if you get the same answer We don’t even really know that But that’s something one could start to do Every time there’s a big post-marketing study of an anti-platelet drug, do the same thing in your environment, see if you get the same answer – So, Bob, I disagree with you There is a lot of evidence to suggest that results from RCTs and results from observational studies actually match up more than they disagree There are very well-known cases where they don’t – [Bob] I’ve seen the Ioannidis things – I know very well-known cases Having said that, I think the issue is about the corpus of evidence You never rely on one clinical trial You should never rely on one observational trial If you have four different observational trials done in four different populations, using somewhat different methods and they all directionally say this is true, I gotta tell you, that’s as good as evidence-based medicine’s going to get – [Bob] Maybe, unless there’s some inherent bias in it But you could also do what I said Do your epidemiologic studies where you actually have a study showing you the answer See if you get it – We may not fully resolve this It does seem like there is some agreement, though, on best practices, good hygiene around how to do the observational studies, prospectivity, transparency and methods and all I thought there were a couple of questions maybe over, okay, right here And then Sean, and then I think we’re going to be out of time – I just want to pick up on Bob Campbell’s comment I’m Bill Crown, I’m the Chief Scientific Officer of OptumLabs and we’re actually trying to get a project going in collaboration with MRCT and also working with Duke-Margolis on this to replicate a whole series of clinical trials The inclusion-exclusion criteria of the trials, seeing if we can estimate the same average treatment effects and then open up the aperture and see who was actually treated and look at treatment heterogeneity and under what conditions can you replicate and under what conditions can you not? Really would be very interested in anyone in the room, or in an extended group that’s on the phone to get in touch with us if they’re interested in learning more about this and participating – Great, thanks very much, Bill Sean, and then I think that’s probably going to be, Sean, you’re probably our last question for this session Or comment – Sean Tunis from the Center for Medical Technology Policy Another great panel, several people on this panel and the earlier one referred to the importance of outcomes that are meaningful to patients being increasingly emphasized by PCORI Patient-focused drug development everywhere The recognition is that looking at benefits and risks of therapies is going to be much improved by focusing more on gathering that sort of either patient-reported data or other measures that are the patients say are meaningful impacts on life Probably people are familiar, there’s various groups out there, like the COMET Initiative and ICHOM that are trying to develop these kinds of harmonized patient outcomes sets I’m just wondering if anyone can make the link of how can we look towards enriching the quality of real-world evidence or evidence generated through clinical care where we might actually have some meaningful and consistent outcomes that are meaningful to patients and not be limited to what’s easy to collect But what actually is meaningful – Well, that’s come up on both panels As a real reason to pursue some of these RWD efforts Any quick comments? I see a head nodding – Well, I’d just say incentives matter Let’s just say if you put that as part of our USDs rankings for hospitals and health systems, I know we at Duke would really pay attention to it (Mark laughing) (audience laughing) And I think that’s the case for almost every health system So how do you actually change incentive structure? – So I would just comment, Cleveland Clinic instituted years ago in their rehab thing area, collecting a patient-reported outcomes questionnaire And it was well taken up because it actually

helped the doctors take care of the patients Though if you do something that helps a doctor do his job, or any healthcare practitioner, do their job better, there’s incentive for them to do it When you tell people to do things and they don’t see how it’s going to do it, then you end up with what happens when you just layer in electronic medical record on a system You don’t reorganize how you do medical care You end up with other errors because people are just trying to get through the EHR So I think there is fatigue among practicing physicians about entering all the data in EHRs There’s fatigue among physicians about all the reminders and popups that they get to tell them to do things I had a doctor a couple years ago who I went into see, and it’s not my doctor anymore He’d ask me a question and then A minute and a half would go by, then he would talk to me again We have to re-engineer how healthcare is delivered The EHRs of today are an enormous step forward, but they are not the final place where we need to be Some clinics have gone to the fact that they have someone who just records the data and the doctor doesn’t have to sit there and record the data into the EHR I don’t know if that’s economically feasible or whatever And maybe a technology will improve that AI will listen to the conversation and automatically put things into a record But asking for perfect information now or near-perfect information now, will cause as much problems as it’s going to solve in the next five years – All right, did you have a last comment, David? – Yeah, I just had one final comment on this discrepancy between clinical trial data on the one hand and real-world data on the other I would argue that we’ll know we have made tremendous progress when we stop asking why the real-world data don’t reflect the clinical trials and we start asking why the clinical trial data do not the reflect the real-world data? – (laughs) It’s back to that flip the trial concept All right, I want to thank our panel for a great discussion Thank you all very much (audience clapping) Now are going to adjourn for an hour So we’ll be starting up again at 1:20 If you have any questions about restaurants, food in the area, please ask at our reception desk (people chatting distantly) – [Man] Well, you know, I’m from a CRO, but I’m a health economist by trade (people chatting distantly) Some of our trials have been five year study image (speaker drowned out by conversations) if you know how to conduct that in population X, you know how to conduct in population Y – [Man] Right (people chatting distantly) – [Man] Weren’t you a resident for Rachel? – [Man] I sure was – [Man] Nice to meet you – [Man] Yes, I remember A long time ago, that’s right, I still stay in close touch with (mumbles) who I’m sure you know (people chatting distantly) Yeah, I still actually do a fair amount of stuff with (speaker drowned out by conversations) It was my classmate, and he left and came back So he’s on faculty there – [Man] Were you a med student there? – [Man] I was a med student and a resident and I got my MPH there So I finished all that and graduated with the MPH in ’99

and then residency in medicine in ’02 and then left and went to do some fellowship and then the grad school (mumbles) south and northeast (man speaking distantly) About 12 years, I did some informatics work at Stanford and did epi training in Boston, so yeah, kind of went all over, love the Midwest, yes – [Man] So I worked at PCORI (mumbles) – [Man] Oh okay, great (man speaking distantly) So I’m the CO-PI of the PPRN of Arthritis Power and have a demonstration project that’s bringing together three CDRN’s data and five PPRN’s data and it’s a little bit like herding butterflies But it’s going and the demonstration projects Yeah, it’s intended to use the infrastructure for something useful and important So it’s going well, but finding all the hiccups and speed bumps along the way But I think all the demo project’s expected to uncover and unearth (mumbles) in large part with (mumbles) – Yeah – Good, so what’s your role with PCORI? – [Man] I’m the Director at one of the clinics – [Man] Okay (man drowned out by conversations) Yeah, we’ve done a number of things Some successful, some not as successful But yeah, our registry though is doing very well We’ve started from zero and we just broke 11,000 people, mostly with RA and autoimmune diseases Yeah, yeah, so we’ve been making great strides – [Man] I have a question about (mumbles) – Yeah – You talked about (mumbles) (people chatting distantly) – [Man] So hardly anybody ever gets the rolls from Medicare, as you know Now this enrollment rate is for commercial claims, of course it’s super high for reasons that you know well I guess my premise is that there is no one perfect data source, so consent people that you can follow them wherever it is they are, in whatever data system that they may be bound in over time So if they’ll keep coming back when (mumbles) your study visits, fine If you can link them to commercial claims, there’s three or four big players that are going to have a lot of your data You could pick your trials sites based on where those sites have lots of representation And in fact, we looked at that for the Zoster vaccine study We’re going to pick our trial sites because we know we have a high penetration of company X and that we can keep following people But it’s really with the premise that there’s never going to be one perfect data source But if you could bridge across them, because people gave you some identifiers and consent at time zero, you had one or at most two clinical visits, got their permission to follow them over time and whatever data source you have access to To me that’s the way to go Because then you don’t have to choose this data source and then they change jobs and change insurance and we lose them But you can really follow them in a much more continual and longitudinal way That is really helped by consent and some identifiers at the outset – [Man] Yeah, well it was good to see you – [Man] Nice to see you (people chatting distantly) – Oh hi, how are you? – Nice to meet you – [Man] Yeah, nice meeting you – [Man] You’re still good to talk at 2:30? – [Man] Yes, so do you want to just pop out to where the waters are and stuff like that? All right, all right, good, great – [Man] Great, I’ll catch up with you then – [Man] Okay, thanks (people chatting distantly) – [Man] I did the breath check

(people chatting distantly)

– Nice to meet you (people chatting distantly)

– [Woman] (mumbles) Nice to meet you, all righty

(people chatting distantly)

– [Man] We have like five more minutes

– [Woman] Okay, all right

– [Mark] All right, I’d like to ask everyone

to head for their seats

We’re going to try to start again

in just a couple minutes, thank you

(people chatting distantly) – [Man] I’m going to be at the far end – [Woman] We’ve been assigned seats – [Mark] All right, good afternoon, everyone I’d like to welcome you all back to this afternoon’s session of our real-world evidence event today For those of you who are joining online, just a reminder that if you have any questions or comments along the way, you can send them to us at [email protected] That’s [email protected] and we’ll get those questions or comments into the discussion if we can And I hope many of you are following along on Twitter There’s been some good discussion, Just use the @dukemargolis handle and #RWE So for our session this afternoon, we’ve already talked this morning about many of the challenges and issues in developing real-world data and applying it in fit-for-purpose uses for real-world evidence development And as you heard, real-world evidence is already in development and use Including in some issues related to regulatory applications What we really want to focus on in this session is some of the industry perspectives, some of these perspectives from the companies, the organizations that are actually involved in the development of real-world evidence And in a position where it could actually be submitting such evidence for label modifications, for post-market safety information Some of the topics that have come up earlier Even potentially for practical clinical trials or other types of real-world evidence in the pre-market setting So we want to hear about how that experience is going with an eye toward some of the scientific or infrastructure or cultural or other barriers or issues that might get in the way of continuing progress on real-world evidence development and use And we’re very pleased to have four panelists with us today that have some extensive direct experience related to real-world evidence and related to regulatory applications Amy Rudolph is the Vice President and Head of US Pharma Health Economics and Outcomes Research and Early development at Novartis Pharmaceuticals Jacqueline Law is Vice President and Global Head of Real-World Data Science at Genentech Symantha Melemed is the Global Product Team Leader at Eli Lilly & Company And Joanne Waldstreicher, Chief Medical Officer at Johnson & Johnson So as in our previous panels, we’re going to start out with some opening comments from our panelists and have a bit of discussion This is a panel where we want to make sure, I think we’re going to have a relatively short comments We want to make sure that we’re hearing from you all about issues, concerns, opportunities that you see in these real-world evidence applications for regulatory purposes

So Amy, please go ahead – Just to start, thank you so much for the invitation, Mark and colleagues Good afternoon, I’m Amy Rudolph, as Mark mentioned We took a little bit different tack, we wanted to have more of a practical discussion So I don’t have the clicker, maybe if you could jump to the next slide What we thought might be useful is to share some perspectives on some key tenets to consider as the guidance development is underway I think we call agree on these tenets And they were elegantly captured in the white paper and throughout the discussions today We talked today a little bit about the environment, the complexity of the environment demanding new evidence I think we can all get behind that That really spans the entire data continuum I think when we think about data collection, we’ve talked today a little bit about reproducibility of data, we’ve talked about some primary and secondary data sources and the fact that we need to bring together data compendiums On this, I’ll note that we haven’t talked a lot, although it was captured in the white paper and maybe we’ll touch on it is, one of the neat things about capturing real-world evidence and using that to compliment existing trial data sources is it can be cost savings A lot of this discussion comes up around pragmatic trials And certainly when we think about the need for large data compendiums, we don’t want to lose the fact that we could save costs So we want to control the compendium but have a robust compendium for offering data forward In terms of tenets, again, these are just, it’s certainly not an exhaustive list I think we could all agree to that But just by way of what kinds of elements might be critical for the guidance to include and likely will Certainly bias management has come up One thing of note and it’s been touch upon today, but just maybe to state it a bit for the record that I think we all recognize, but again it’s important to state that bias mitigation cannot be absolute And I would argue and I think others would as well that bias mitigation can never be absolute But I think that’s important to keep in mind and we’ve talked about EHR and some data, missing data and some other challenges We can mitigate the bias, but not completely I think we need to, if we can, set boundaries for what is good enough Marc, you talked a little bit about that, but also around what kind of primary versus secondary data sources would be acceptable And are there boundaries for what’s good enough? And what is outside of those boundaries? If we can get some information around that I think that would help us all try to bring forward the best robust packages that would fit within those boundaries We talked a lot about database suitability If we can have some directionality around that, if there are databases or approaches that are just not suitable, again, it’s around the boundary question I think we should state those as clearly as we can And the last point is around patient-centric data This was raised, it was touched on previously by David The concept of the collection of lots of different types of data Adrian talked about his mom and her experiences Sally, great discussion around fatigue, I thought that was a supreme example of the different types of data And the use of sensors, we’re becoming more and more intelligent about how do this in an effective way But I would argue nationally, we don’t know how to handle big data, sensor data For example, I think the value framework organizations are struggling how to integrated PROs and even sensor data into that framework I think it also is applicable here that we’re all kind of struggling how to do that But if we want to have an intimate look at the patients’ lives, this is one way to do it using different types of data and we need to be able to fully integrate that I captured here adherence/persistence also, fundamental as well to the patient experience Turn it over to you – Thank you – Yep, thank you Yeah, thanks again also, Mark, for the invitation and have the opportunity to share industry perspectives here Next slide, so I just have one slide I think there are many, many factors that are driving the interest and opportunities of using real-world data to really support a broader healthcare decision-making First of all, it is the improvement in the quality of real-world data The improvement in the availability as well as the speed of these data become available And also I think this morning we heard about maybe the relevance of the data Clinical trial data look at certain endpoints, but in the clinical setting, some endpoints may not be relevant any more So I think that’s fundamental, in terms of driving this trend forward But also with the advances in medicines, diagnostics and technology, in the area that we’re developing personalized medicine, precision medicine The way that we develop drug need be very different than the traditional way of drug development So coupled with the increasing drug development costs and timeline and also the pricing pressure

I think all these different factors really drive the industry to think about how to use real-world data to inform decision-making So we’ve heard that there already opportunistic use of real-world data to information regulatory decision Including the rare disease setting as well as the post-marketing safety surveillance But how we move forward confidently of using real-world data to inform broader healthcare decision-making So there’s some ideas how we can move this from a concept to practice So first of all, we heard a lot about data standards and how we can really set the standards of collecting data, what kind of data need to be captured Also the quality assessment and the requirements To Amy’s point, how good is good and what are the boundaries? We’re not expecting real-world data to be monitored and maybe not as clear as clinical trial data, but does that matter? Is that going to influence decision-making? And then endpoint definitions, definitely is a big topic in real-world data It’s easier said than done, in terms of defining the relevant endpoints in the real-world setting And it requires a lot of collaborations across the industry with FDA and academia to really think about what is really the relevant endpoint when we look at the data from the real world And then the other thing is how we can protect data and patient privacy I’m so glad that this morning, a lot of conversation on the governance So it’s really fundamental for us to protect data and patient privacy So one of the requirements when we look at real-world data, is there going to be a informed consent required? What are the HIPAA requirements, et cetera? These are the things that need to be defined and clarified And then going down the line, when it comes to submission, we are all very familiar with the submission requirements data package required for clinical trial submission So in the real-world setting, are we going to think about audit? Where there’s a source data verification requirement So these are the more practical things that sponsors would need to think through from the very beginning of the clinical development, if we go down that path of using real-world data to support regulatory use And I think one thing that is really fundamental in terms of this industry is we need to change our mindset Using real-world data to support regulatory use is an innovation and driving innovation, we need to foster the right environment and culture So right now there are a lot of hesitations even just to think about this path because of the uncertainty and also the potential impact on timeline and the risks that are going to come with using real-world data So having the opportunity to have early input from FDA, interactions with FDA on the drug development program using real-world data would really help remove some of this uncertainties and engage different stakeholders to have this conversation together And then finally, it’s to really have the opportunity to have pre-competitive sharing of use cases Again, we are in this journey of driving an innovation in this industry So finding ways that we can learn from each other, be it a successful use case or a failure use case, I think we can all learn from how to move this forward – Great, thank you – Very good I don’t have any slides, but just a few things to add on to what my fellow panelists have so eloquently described Some of the areas that we’re really interested in using real-world evidence is certainly on the regulatory side, but then also moving it earlier into the lifecycle program of the drug When we’re making these trade off decisions between a traditional randomized clinical trial and a pragmatic or observational study, the more clarity we have in terms of where that will fit in to our submission activities, global registrations and other clinical development for the compound will really help as we look at developing these compounds Some of the areas that we’re really interested in is around using real-world evidence for dosing, for historical controls, even understanding biomarkers as we’re starting to get these high-quality linked data sets, where we feel good about the quality of the biomarkers that are there and the level of detail the laboratories that are performing the analyses Potentially using it for control arms for rare populations or in oncology, where I spent the majority of my career, using that where we have single-arm studies to understand how the drug’s performing in a rapidly evolving scientific landscape For post-marketing commitments and then of course for line extensions, which has already been extensively discussed So as we start to think about the data ecosystem that we would be bringing forward, we certainly remain committed to randomized clinical trials, but what we’re looking forward to is guidance from FDA and then as this room and people on the web as well evolve the landscape, is the data ecosystem that will exist between observational trials and pragmatic clinical trials And then of course from the randomized clinical trials

that have provided the foundation of the majority of work that we’ve done to bring new medicines forward over the years A few things I just wanted to bring up in terms of barriers, as I think you alluded to it nicely, is that this is an innovation And we have generated a group of folks that are used to doing randomized clinical trials And so as we think about the regulatory science, How do we do the submissions? How do we do audits? I think the people and processes will need to evolve at the same time And then also, I just wanted to call out that we do, we believe, have a unique opportunity in oncology, we’re certainly interested in real-world evidence across the entire spectrum But I think in oncology, we have key stakeholder alignment We have a very engaged community from the advocacy community to the ASCO, and other unifying bodies From pharma and others, we have some really great examples of high-quality data sets that we can use And the rapid pace of change in the space, I think we all feel it, whether or not you’re in oncology or not, it’s unbelievably quick So I think we have a really interesting opportunity there to look at pilots and others moving forward So I think we’re excited to get started and we look forward to the conversation today – [Mark] Great, thanks Symantha – Thanks so much for the opportunity I think what we saw in the white paper, which was excellent, and also from many of the speakers today is that rigorous real-world evidence really helps provide insight into questions that are difficult or infeasible or cost prohibitive And can really help drug development or post-marketing, from both an efficacy as well as a safety perspective I think what we have the opportunity, though, as an industry to really help contribute to the goal of a learning healthcare system And I think this is where we focus and where we want to focus from a public health perspective and set the bar higher for all of us And I think we would all benefit as a public health system if we do that So the themes that I want to talk about were being more transparent with our data, our analyses, our methods and our results And really trying to think about the learning healthcare system I was given the assignment, I was volunteered, to talk about safety today, but I don’t want to let the opportunity go by without mentioning a couple of points about effectiveness I think you can’t say enough about the potential for predictive modeling It’s great when we can use real-world evidence and real-world data to understand the generalizable benefit and risks, but really there’s incredible power to come if we can use predictive modeling Both from traditional sources of observational data, as well as new sources that we have and that you all know about Patient types of data, people types of data, that we can then put in for predictive modeling Because it’s all about, in the learning healthcare system, it’s all about maximizing the benefits and minimizing the risks Right now it’s difficult for us, even if we believe that we have good predictive modeling, it’s difficult for us to be able to discuss this, and we’d love to be able to work with regulators to be able to do analyses that meet the rigor that they would agree with and that we could then talk to healthcare providers and patients to be able to maximize the benefit and minimize the risk I also don’t want to let the opportunity to go by, and this was something that was raised by Laura Esserman, we really believe strongly in platform trials and master protocols I think the more that we can do to have similar protocols or the same protocols, the same infrastructure, so we’re evaluating products on the same basis with the same endpoints and the same rigor Even if products can’t be evaluated at the same time, but if they can be put into master protocols or platform trials as they become available, I think we all benefit from a public health by doing that And of course, it raises the bar for us as industry, but that’s a good thing It forces us to go higher and better and for more transformational therapies And I think everyone benefits So I wanted to talk about safety In terms of safety, I think that we could think about a future where we could work together to further understand the safety profile of our products Going even farther, but again, I want to keep those themes in mind of transparency, inclusiveness, using the best available methodology and analyses So as we all know, the safety of our profile is driven by clinical trials, of course, especially pre-approval But of course we have them post-approval Then post-marketing adverse event reporting or spontaneous reports, as we call them And then if a signal is observed, an observational epidemiology study

And for retrospective observational studies, rather than randomized studies, the white paper says, and Marc Berger stole some of my thunder, so thanks for that, (audience chuckling) talked about the importance of pre-specification of the analysis plan, the strength and diversity of the real-world data and the reproducibility across different databases This is something that we could not agree more with We think that there could and there should be right now, much more rigor in insisting on pre-specification And I would go further and say even pre-registration of these types of protocols and analysis plans for observational studies Protocols for observational studies can be registered on We have done that, they can be and they should be registered And although the system is set up primarily for clinical trials, there’s no reason why we, journals, others, can’t insist that protocols be registered before they’re done There also a registration site for meta-analysis protocols called PROSPERO We also have to be sure of the strength and diversity of the data And there’s no reason why we shouldn’t insist on multiple databases and evaluating the reproducibility and the rigor of the data that’s generated from real-world evidence studies I think we could go even a step further As I said before, the sequences, putting aside clinical trials, of course, which is very robust for safety signal detection We now live in a world where we have spontaneous, as we all know, underreported adverse events as well then looking at observational data So I’ll start with the observational data Observational studies, like Sentinel, which has been incredibly valuable for understanding safety and for public health Now as we’re moving forward and we’re getting more experience with these types of analyses and we get access to even more databases, not just in the US, but globally, worldwide, and we’re sharpening and developing our methodology, might there be an opportunity not only to wait until we see a signal to do an observational study, but to proactively look at observational data for signal detection in addition to spontaneous adverse event reporting? We’re looking at methods to do that To proactively look at observational data and mind you, when we do that, we see not only our own drug data, our own treatment data, but we see everything, all the different treatments, all of the data Is there a way in the new world that we’re living in to proactively look at those data to generate signals Which could then be confirmed or refuted by doing an epidemiological study, such as Sentinel Since we’re talking about observational data and we’re talking about safety, is there an opportunity now as we’re getting better understanding observational data and I love the discussion from Laura Esserman about seeing and looking for adverse event reporting within their health system As we’re doing more of that, can we revisit spontaneous adverse event reporting? Which we know is severely underreported and not perfect Of course adverse event reporting is critical and will continue to be critical for years to come to understand the post-marketing safety of our products But since we’ll see everything in real-world data, and we can get better and better over time, what can we do better or differently? And I would throw out, what can we even stop doing? For example, at our company, we literally spend millions of dollars and I counted, we have hundreds of people working on non-serious spontaneous adverse events Not just reporting them and entering them into a system, but reviewing them and auditing them, reporting them, summarizing them, getting ready for health authority audits There are hundreds of people looking at that I just asked my group over the past couple of weeks to look at a few large products, just as an example And I asked them, were there any label changes, based on non-serious adverse events in the past few years? Now of course, this is not comprehensive And I’m sure that there are label changes all the time, but in this recent review that they did for me, there were no label changes based on non-serious adverse event reports from spontaneous reports So the litmus test that we should take forward is are we advancing public health by all this enormous effort that we’re devoting looking at non-serious adverse events? Something to think about in the future, whether we could look at observational data, health data, data in the University of California system Other ways, PatientsLikeMe, other ways to look at non-serious adverse events which would be better from a public health perspective We could also think about looking at serious and even designated medical events

The adverse event reporting system has played an important role, I would say critical role, in defining the safety profile post-marketing But could we do even more and even better? As the learning healthcare system, in addition to spontaneous reports? Could DMEs be flagged? I know everyone hates flags, but if anything’s going to be flagged, could those somehow electronically, all of them, be captured so that we immediately in real time get access to the broad amount if information surrounding those designated medical events and get better follow up and more complete reporting of events? And I would say maybe even faster reporting of those designated medical events so we could label them earlier and prevent more harm in the future And be much more efficient and complete, again, taking that public health lens So just putting it all together, my hope is that we can work together over the coming years to define how we can be more transparent, work together in a more open way We’re getting broader and deeper into real-world data And as I said, we see it all As my group looks at the real-world data and looks to understand our product, we see the data with all the products, all of the outcomes Is there a reason we need to do this separately and in silos? Is there a trusted third party or a public-private partnership we could work with together to look at all the data together to generate signals, to understand and evaluate signals To do this together on behalf of public health Could we do more predictive modeling to maximize the different, minimize the risks, then revisit the data and see if we’ve had an impact Have we decreased the adverse events that we’re seeing because we’ve stopped using the drug in certain types of products who have the highest risks? And we see collectively if we’re accomplishing our goals as a learning healthcare system Could we even pre-define our protocols, post them and actually make the source code available publicly? Could we do multiple types of analyses as we do one protocol, do many sensitivity analyses and many different types of analyses Make it all available publicly And as I said, do it all together to empower the learning healthcare system Thank you – Great, thank you all Clearly, as we’ve heard earlier today, a lot of potential for better evidence In some cases, new kinds of evidence or potentially transformative evidence at a lower cost than what’s been available with existing systems and even existing uses of real-world evidence to date But I also heard in the comments, some concerns about uncertainty about how to proceed and the risks of incurring additional costs on top of all of the existing regulatory requirements And I want to follow on that a little bit, to try to take us again to some practical steps that will get us to that longer term vision And a number of you mentioned areas in which more clarity, more guidance from FDA would be helpful, like around submissions and like around early input Although, other Jacqueline, Jacqueline Corrigan-Curay, mentioned that FDA wants to hear from you early, about real world uses So we’ll come back to the FDA issues, but I want to start with some of the issues that Joanne and others mentioned around what industry can be doing differently to help Or what industry needs to do differently to help hasten these efforts along And maybe you could start with whether you all see any cultural, or are they more technical barriers, to some of the recurrent themes here? Transparency, prospective definitions of protocols Something that’s done a lot in the clinical trial context, but maybe not so much here Reproducibility, via data sharing, steps like that Those are not the way industry practices work today What’s standing in the way of that? If you all seem to agree that those are important steps for advancing real-world evidence? – Maybe I’ll start I think, and please add perspectives, there is a tremendous amount of rigor that happens and it’s just not made, it’s not shared, it’s not exposed The what I’ll call internal rigor around making sure it’s pre-specified and we all have forums that do this, work in a very collaborative, cross-functional way What a step forward would be that we invite others in, from outside of our bubble, I think, and make it even more transparent So it’s not that these things are not happening, I would offer In terms of transparency, pre-specification, but it’s not happening in a broad way And it’s maybe not happening in a way that is fully collaborative I don’t know if you guys would – Yeah, I agree very much with you I think rigor, definitely is a key component to really build trust internally and externally

However, there’s no platform or opportunities for us to broadly share that and how we are thinking about rigor and quality So I think that would be really one step that we can take forward to share our experience, our methodologies and bring all these together – A lot of that, it can be done pre-competitively Because it’s a specific research question around the specific medicine that ends up being the competitive part of it But a lot of the sharing and standardization and all of that can be done in a pre-competitive space – So let me just add a couple points I think we face the same cultural issues And when I say we, I don’t mean just industry, I mean sponsors or people or groups that do clinical trials It’s not just industry We face that with clinical trial sharing, as was mentioned earlier And we’ve found a way, we found a path to overcome that We started working with trusted third parties We work with Yale and YODA and many of the companies work with CSDR with WELCOME and Duke So we found a way to start doing that It all started with registering clinical trial protocols And as many of you have seen the publications, that wasn’t widely done until it was required by the journals, by the medical journals As soon as they set the guidelines that they won’t publish a study until studies are registered on, there wasn’t that much and then there was the hockey stick up But we found a way to do it We found a way to post our clinical trials And we are also are now finding a way to share our clinical trial data Getting around the issues and working very carefully around consent and privacy, et cetera So I think that there is a path forward and we can work, especially after conferences like this and some of the meetings coming up with IOM, there is a path forward to do that And we can start, but it doesn’t just require us to do it I think it requires other colleagues in academia and other parties that do observational studies We have to all be in this together – I think there needs to be this mindset, is not there yet And we are all learning together We are sharing experience and methodology, not because everything has to figured out The data quality is not all there yet But we are sharing that so that we can learn together along with the data providers and others And okay, how should we improve so that we can eventually get there? – Yeah, it sounds like a great time to add to the platforms for supporting this area I like Joanne’s analogy to what’s happened with clinical trial transparency and rigor over the years To build on this, several of you mentioned this notion of platforms So Joanne talked about it, fairly extensively, around post-market safety active surveillance using observational data on pretty much everything, it sounded like, from your comments But Symantha, you mentioned other potential platforms or use cases, too, so I wonder if I could push you all on some additional specific examples Maybe more on safety, if you want to expand on that, Joanne or others But Symantha, maybe I could start with you You mentioned that oncology might be an area that’s primed to do this, could you say more about what this kind of pre-competitive platform approach would look like? It sounds like it fits a lot with what Laura Esserman was talking about this morning – Yeah, absolutely There are some advantages in how data is collected in oncology that has helped us move quickly One of the things that certainly from that that we’re interested in is the amount of biomarker data collection that’s now routinely being done in oncology And as we think about using real-world evidence, not just to register products on the back end or in the later part of their lifecycle, start to understand patient populations that they work better or don’t work as well in And do that based on the high-quality biomarker data that’s being collected at the same time In addition, we’re often relying on either making big investment or clinical trial decisions or even making decisions to go to regulators with single-arm clinical trials where we see big effect sizes relative to what we think of as historical controls Of if we’re seeing a biomarker that’s in a very rare patient population, where traditionally you would be testing 100 patients to find four, are these tests are routine enough and that we feel confident enough in the quality of them that we can go and find some of those patients and start to see the benefit there Not to double-click on something that I’m a big geek about, but understanding then how that 4% of patients performs relative to historical control when the field’s rapidly advancing really matters So we don’t know if we’re seeing a big advantage,

because prognostically they’re better, as they often haven’t been studied And there’s often ways that we can do this where we pre-specify So I think in oncology, we just have a massive amount of relatively high-quality data that we’ve got some really good curators that are working on, it’s not perfect We share all of the same concerns that were keeping folks at night in the morning But I think that there’s some real advantages that oncology provides right now – Some other platform use case opportunities that you all would like to talk about? Some other specific opportunities? – Well, even in situations when we are not able to do a clinical trial, I think then how can we leverage real-world data to identify the patients and maybe also looking at off-label use and seeing where the activity is and treatment benefits are And the order and biomedical needs Or use real-world data to create some synthetic control arm to, as a contemporary controllers Symantha mentioned that the field evolves so quickly So looking at literature and historical control would not be relevant any more So I think these are many uses of real-world data that we can think of in the practical drug development – Let me extend it to more common situations Someone this morning mentioned the diabetes medicines which require now cardiovascular outcome trials As I look back several years, I think, I can’t even count the number of companies that have done their own randomized placebo controlled cardiovascular outcome trial, which is great But in the future, might it be also good or instead, could we work together through a trusted third party that runs the study? So we’re looking at multiple products in the same study with the same endpoints and the same rigor and potentially even looking at combination drugs If you think about taking any of those drugs that have shown cardiovascular benefit for diabetes and now doing a combination study to see if the combination is better than each individual alone for a cardiovascular outcome trials, you’re talking about, I’ll be on Medicare I told someone this morning, I’ll be on Medicare by the time that’s done And it’ll require hundreds of millions of dollars of investment, et cetera And isn’t it best for everyone if drugs are compared in the same trial framework and we can right off the bat look at them together and also separately – So what’s getting in the way of progress on these kinds of platforms? It sounds like some of this is happening now, you got the ingredients What would accelerate progress? – If I could add? – Uh-huh, please – I was going to add that from a platform perspective, I’m going to take it up a level You kind of touched on it This is actually happening There’s advisor groups, PCORnet advisor groups We’re all coming together I might argue, though, it’s still a closed system Why can’t we pull more patients and payers and other individuals within the ecosystem, to borrow your word, why can’t we pull them in together? And that’s happening a little bit, but I think we need to do that in a much more concerted way Because I think to your question specifically, that is what’s going to take us to the next level Where you’re tackling big health care challenges together, pulling folks from the FDA together with the table, with us, with patients, with the payers in the system so that we can all tackle it together And meet the collective needs of ideally the patient first, but then all the other needs as well – [Mark] Other thoughts on acceleration? – I do think regulatory clarity is a huge piece I know that that’s, I mean, reading the title of the meeting, it’s a little bit like Captain Obvious But I do think for us as we make trade-off decisions And there’s a lot of excitement and there’s a lot of enthusiasm and there’s a lot of interest in finding what pilots do we invest in? How do we work? I think this is an area that is really neat, I worked in the biomarker space as well And I feel like there’s a lot of collaborative spirit and a lot of energy, but we’re trying to figure out where’s the best place to aim it? So that we can bring our medicines forward faster and better? And I think when we start to get some of that regulatory clarity, and even hearing like, say, hey, here are our ideas, having FDA say, “This is what we’re interested in as well.” And then having that say, okay, now as we start to register a medicine, this is how we can make that progress that will really help I had promised we would get to a discussion of opportunities for regulatory clarity And having been on that side of FDA, there are an awful lot, I mean, this is complicated stuff – Of course – Health care and medical products, so there are a lot of areas where FDA could provide more guidance or input or feedback Anything that you would like to particularly highlight? I have some useful items, Jacqueline, on your slide, that went into the tenets that Amy described

One of the things that you mentioned there that I would like some comments on, but again, feel free to raise others, is this issue of how much bias is tolerated? Or how much data imperfection and result? Consequential result imperfection will be tolerated? That’s something that several of you brought up But I appreciate hearing about that and about any other particular areas where more regulatory guidance or engagement would be helpful – Yeah, so maybe I can share our experience working with oncology EMR So obviously, the mortality is not regularly captured in the EMRs, so we do need to find a way to supplement the EMR data to come up with the mortality outcome So at the end of the day, do we expect 100% completeness of the data? Or how much missing data can we tolerate? And really it depends on the context If you are thinking about using that outcome data to support a regulatory decision for a treatment, that depends on the treatment effects Some treatment you would be able to tolerate a higher degree of missingness And if the treatment effect is huge And this is probably the area that we need to go a bit deeper into the technical side on, okay, what kind of treatment effects are we looking at in what setting? So then the data quality depends on the context And I think this is an area that we would need to collaborate across, because this needs to be a common understanding and agreement on okay, how do go about to collect complete data? And also with FDA guidance on that as well and agreement Because it’s very challenging if we do not have clarity or at least an alignment on, okay, how do we approach this problem – And then what advanced statistical methods are acceptable or reasonable? From the statistical, I think there’s the data side and the stats side and they go together about what’s allowed, what’s okay – And I’d just add, one of the things I tried to frame up was I think the idea of reproducibility is key You brought it up as well I think, though, we must consider that you’re going to need a data compendium, but we don’t want to lose sight that one of the really neat things about real-world data and translating to real-world evidence is that we can save costs and time And we don’t want to lose those elements to get the patients who need the therapies quicker, we don’t want to lose that in the fact that we’re building and building and building a compendium so we have all the right elements to come forward, what is good enough, in terms of missing data? For example, the VA is a tremendous database It’s outstanding, if you are particularly interested in heart failure, as we are, and committed to this space and those patients, you can get echo parameters, extremely deep data The gender diversity isn’t maybe what we would like, understandably, in that data set So we need to look to other data sets, because you’re not going to get everything out of one source But at some point, how many studies do we build into the compendium and where are the parameters to address the missing data questions, the bias questions? – And then I guess the last thing I would add is that what endpoints are relevant for individual therapeutic areas? Because you can build a great data set that is as good as good can be You can start to say, okay, these are the statistical methods that we feel comfortable with and that we can use to analyze this data And this is the level of advancement we can go to But then we also then need to say, for each individual therapeutic area, what are the endpoints that are relevant to make decisions?> But then also, hopefully for registrations or label enhancement purposes, plus efficacy and safety – Great. I would like to, I said for this session, we want to have plenty of time for comments and questions from all of you here and those of you who are joining us online around how we can support real-world evidence development and accelerate effective regulatory uses So I’d like to open this up for comments and questions at this point So anyone, I see a hand up in the back, (man speaking off microphone) Okay, hang on one second, we’ll get a microphone to you so everybody can hear – [Tim] Hello, I’m Tim Lance with Century Data Systems So I’ve heard I think everyone on the stage in a couple other sessions talk about the importance of reproducibility when you’re using real-world evidence And I wanted to ask the question, how do you see reproducibility playing out, given the fact that data sets themselves are very inherently different? Can you actually develop a method that’s going to be reproducible given the heterogeneity of the data sets that might be used? – I think it’s one of those answers that you’re going to hate, because it all depends

I think it’s important to look at different databases to ask the question Then if you get different outcomes, then you look at it to try, that’s only the beginning You do a study, you get results and then you have to understand them So is there a reason? Is it because the Medicare database has patients that look like this and the VA database is only men? And has a certain kind of patient population So I think if you do get a different result, it adds to the richness of what you need to understand in interpreting the data But I think those are important steps Understanding the diversity of the response and understanding the diversity of the results, I mean, can really add to your understanding and adds to the richness of the data that you get – And what’s your sense of the state of the science in this area? It seems like kind of an extension of meta-analysis and are we really at a point where we understand how different, whether it’s claims data properties or different other key features that data do, predict or influence the results that you get? – I’ll let others respond as well, but I think it’s an evolving science I think we’re getting much better with more experience and again, I think we have a lot of great experience Especially with the Sentinel experience, the ODYSSEY experience and others And it’s something that I’m sure will be evolving over the next couple of years – Yeah, I agree on that the databases could be very different But I think because they was set up for different purposes So now that we actually have a goal to use these data to support regulatory decision, or some support some sort of healthcare decision-making So if we have that common goal, actually we can help shape the database to somewhat more similar They collect similar data and use similar methodologies But that definitely is going to be a journey and require a lot of collaborations across – I would just add that I think there is a call to action to have more national fully linked data sets, we have so very few But we have such a diverse nation and forget about if you want to have something that’s globally applicable, that’s really difficult But speaking from a national perspective, we have such diversity, we must challenge ourselves to develop additional data sets or this evolution will not be as rapid or as complete, that we’re all talking about, as we want – Other questions or comments? Yeah, over here – [Ritesh] I’m Ritesh Jain, EMD Serono Since we are talking a lot about collaboration, I just wanted to make a comment about TransCelerate, which is a platform where member companies across comes along and I think they have an interesting work stream, which is working on collecting placebo data from controlled trials across different therapeutic areas I think that’s an interesting work stream that’s going on – TransCelerate has done this pre-competitive work in a number of areas And it does seem to fit – Absolutely – with some of the goals here – Yeah, it’s certainly a model for working together pre-competitively I would say there’s also a place for trusted third parties to be a convener for situations like this and that’s also extremely valuable – Question, yeah? – Hi, thank you so much for the panel And for taking my question With respect to the third parties – Oh please be sure to tell us who you are, too If you don’t mind – Oh, I’m Sonia Pulgar, I’m with Ipsen Pharma I was wondering if you could comment on the oncology space, for the role of cancer research networks and maybe for us to collaborate with respect to sponsoring and supporting cancer research networks? There’s one I think that’s quite active already – I think that as an oncology person, I guess I can start But yeah, absolutely, and I think we’ve seen on the evolution we’ve talked a little bit about when clinical trials, when we see this hockey stick, I think the sharing of next generation sequencing data, when we went from TCGA to what we’re seeing now with GENIE and real broad sharing of data there And how quickly that that has gone, I think that’s been really helpful to see that happen in a positive way So I do think yes, absolutely, that some of the bigger cancer research network collaborations would be definitely be a way to go and we’re very interested in that – And I think especially for platform trials in the future and I think it’s not only going to be the cancer networks and the physicians who drive this, but I think in the future, patient networks and patient groups will drive this And of course it’s in patients’ interest, people’s interest to think more about platform trials rather than just setting one individual product at a time So I think that’s the future

– I think medical associations, I would add, as well That’s happening – Yeah, absolutely – And there’s much more sophistication, but they’re really a fantastic partner to pull everybody together It’s a really effective way – Yep – Thank you – [Juliana] Hi, I’m Juliana from Deloitte Consulting And I just wanted to add onto that, and the conversation that you had just now sort of leads into this I was also thinking about disease advocacy organizations and the role that they would play in terms of collecting and aggregating the data And not just the biomarker data that you’re talking about, but also a lot of these patient recorded outcomes and the patient experiences both of being on the drug and also some of the less quantifiable outcomes that we we’re talking about in terms of time spent with children and time spent being happy instead of miserable All of those kinds of outcomes that so hard to get out of an EHR and that are so critical to people’s lives and how they feel about a drug And just not readily available But being able to bring that to scale in order to have the statistical power that we need to be able to actually draw some conclusions based on that – Yeah, I think patient advocacy groups are growing in their sophistication and ability to do even their own analytics, I think But I think the beauty in there comes with partnership Because it depends on how do want to you specialize? And they’re so incredible about especially coming with a patient voice and bringing that forward Do we really want them to be the specialist and building the analytics and doing that? I don’t know that that’s the fastest and best solution, but making sure the partnerships are there And again, from the platform concept in terms of bringing people together, I think that’s where you really want to bring them in And again, have the patients at the table as a builder, from idea forward – Hi, Jason Harris with the National Health Council I’m just building off of that About a month or two ago, we hosted a round table, just on that the patient perspective on real-world evidence so just thinking about the different skills and tools they would need in order to really get engaged So I’m just curious what are your guys’ thoughts on how would that help you in your industry, if you can some of those views incorporated into what they need and they can have those skills to then – We’re looking into that question and Jason, while you’ve still got the microphone, I know National Health Council recently had a new report come out related to steps that patients can take, or from patient perspective, would matter for real-world evidence development Do you mind expanding on that a little bit, too? – Yeah, absolutely, thank you, Mark Actually we just recently released it today in alignment with this meeting – Very recently (laughs) – It’s a white paper on the patient perspective on real-world evidence So we went through just the basic concepts of let’s get a standard definition, similar to what we’ve heard today And then thinking about what do patient groups and individual patients need to know about real-world evidence? Whether it be some of the data concepts or the different terminology that comes from it And then starting to think about the skillsets So talking about registries or what do you need to engage in trying to develop things like that But certainly, check out our website, happy to share it, it’s about not too long – Thank you, comments? – Yeah, I guess one idea I have is that maybe step back from real-world data and just even think about clinical trial data Or clinical trials, how can we actually follow up the patients after they are off the trial and then that could be an opportunity for us to pilot some of these data collection with the patients and partner with them to really understand what are the relevant outcomes? And importance in data that are being collected after they off the trial that can help inform their decision on the medicine And also on the health studies And that could be maybe an easier pilot also, for us to get on – I think the patient advocacy networks, I think they’re incredibly powerful and their input is incredibly meaningful And I think we’re just at the beginning of seeing the impact I recently had the experience working through the IOM, or in Nasum on, as we talked about clinical trial data sharing And they were thinking about, from their perspective, if they give a grant to a researcher and then that researcher doesn’t share the clinical trial data or the data, I should say, with other researchers who are working towards the same goals of conquering a certain disease that they have, then how do they feel about it? And when it came down to it, they’re thinking about, well, maybe they should have their own guidelines on clinical trial data sharing because they’re giving money and they should set the bar So thinking about that, and this is what we’re all aiming towards in the learning healthcare system is we should hear from patient advocacy networks, from patients, from people, what is important to them And how we should approach real-world evidence,

real-world data, and then be able to try to meet those needs with what we’re developing and working towards – Thank you, back here and sorry, before you start Actually I know there’s another comment Maybe just to help keep us running ahead of the comments, hands up for just a second for others who have questions? Okay, great, thanks – [Michael] Michael Liebman, IPQ Analytics All of the discussion today about real-world data, real-world evidence, a lot of the operational or technology based issues, data sharing I’m curious as to your reaction Just yesterday there was a paper published, or described in Medscape, where pathologists had, a large bank of pathologists, over 100 I believe, had about 1,500 specimens that they measured I believe it was a melanoma, over a period of six months with repeat measurements And the concordance was about 70% for stage I, about 80% for stage V, but about 50% for stage II, III and IV So the reproducibility among pathologists in that kind of study indicates that the real data, real-world evidence data, doesn’t necessarily have the same reproducibility or confidence And I wondered how you were anticipating that We did a study in breast patients where we used two FDA approved markers and saw some much similar effects measuring HER2 with IHC and FISH and finding that again, zeros and IIIs were fairly concordant, but Is and IIs have a very large disparity And that data is not necessarily captured just in the definition in the fields that we’re talking about sharing – Yeah, I’m happy to speak to that Absolutely, as we have looked at biomarker data in the context of real-world evidence, I think the confidence in the testing that’s being done, the lab that it’s being done, the platform that it’s being done with And data sets where you have that level of detail are crucial and I don’t think we can be flip about how important knowing that is, nor should we underestimate the regulatory hurdles that understanding that data would meet before we choose And I do think to me this is something that folks that don’t marinate in biomarker work on a regular basis and think about companion diagnostics and registering drugs to test, don’t necessarily appreciate So when we start to look at these data sets around oncology, the ones that have high-quality biomarker data where we can take the provenance of the marker all the way to the studs And then as we talk to regulators about that, I think that’s incredibly important and it’s something that’s often glossed over So I think it’s a really good point – Yeah, seems like an issue for real-world evidence, but also an issue for real-world care – Yes, absolutely – Absolutely – As well, yes – Hi, this is Fred Yang, KBP BioSciences So we mentioned about the real-world data in the specific therapeutic area We mentioned the cancer, oncology So what other therapeutic areas that would have great use of real-world evidence, especially pre-approval that could expedite the new drug for the patient? I did some antibiotics, I know like pseudomonas or all of those infections are rare and really hard to find And the regulatory pathway, obviously, you can not do the large controlled clinical trial, because it’s really hard to find those patients So is that something the agency and all of the industry leaders are thinking about? – Maybe I could start I think diseases that are largely symptomatic, where the regulatory endpoints, FEV1 was brought up in the white paper, but I’ll speak to PASI, which is a commonly used regulatory endpoint for psoriasis A coverage of body surface area endpoint The challenge is the PASI’s not using the real world, and it’s really not clear to understand from a patient perspective or a provider perspective So really understanding the symptoms of, let’s say, a dermatologic disease are critical

And arguably more important even In particular, when you have a symptomatic disease like that, and more and more diseases that are complex and have increasingly complex symptomatology, I think it’s paramount – Yeah, so I would add that in the rare disease setting, when you have a hard time even finding the patients, I think using real-world data, looking at registries and also in the pediatrics drug development setting, that would be another area that I think leveraging real-world data rather than think about always doing large clinical trials would be another relevant setting One more would be in the neuroscience space and this is an area that depending on the disease that we are talking about, but for example, in Alzheimer’s disease where this a long progressing disease and we are not going to be able to follow up the patients in a clinical trial for 10, 20 years So do we again leverage real-world data to really understand the disease progression and identify treatments that can benefit the patients would be another area that we could use real-world data – Thank you – [Bill] Bill Crown, OptumLabs Just a comment about research design as opposed to methods I think we tend to focus on the methods There’s been so much methods development But actually, my own personal opinion is that the methods are in a pretty good spot now And particularly with all of the work that’s been done in the safety area with Sentinel With the quasi-experimental design, we’ve gotten to a point where the method are pretty well established And the studies really don’t vary all that much with different methods What’s much more important is the study design and how you frame the question So Women’s Health Initiative is a classic example of this And Miguel Hernan and Jamie Robins at Harvard developed this idea that they, a concept called the target trial And it’s the idea of designing your observational study like it was a clinical trial And my sense is that what we’ve learned from the safety literature so far is that when you pick new initiators on drugs, for example, and you have a thoughtful comparator that you’re paring with the intervention treatment that you come to something that is more reliable, in terms of the evidence than if you were to just take all comers or do a less thoughtful comparison So I just want to put in the plug for thinking in a regulatory environment about what’s the guidance in terms of research design? Because I think if anything, that’s even more important than methods – I see nodding – I think we all fully agree – Absolutely – One of the things that’s being piloted in ODYSSEY is that the design, the protocol, let’s say, for studies is done collectively by the group of interested parties and people who want to participate So you get that richness and I think Marc mentioned that earlier You get that richness of input before you actually do the study, rather than waiting ’til it’s criticized afterwards, when it’s published Get as much of that rigor ahead of time and thank you so much, that’s a very important point – Yeah, so here, here and then Bob and that’s probably going to close us out, go ahead – [Audience Member] (mumbles) so I just want to comment So as an assumption that commonly may be not only part of the product It would also be part of a service beyond the product Patients of only programs that they resemble And the patients of any program that’s happening in the real world So it’s very challenge to comment by communal trial settings And it’s very important for us to generate the evidence on the service beyond the product And then also the critical job to communicate this kind of evidence to our stakeholders, patients, payer, regulatory agencies So it would be great if we can have some clarification on how we can generate the evidence, how we communicated that kind of evidence Even potentially incorporating the labeling, so it will give us more flexibility communicating this kind of evidence I think that’s critical – Both on label and it sounds like you’re talking about maybe some off-label communication, too Comments, yeah, okay, agree Thanks, Marc? – Marc Berger, so in prior recommendations from ISPOR, we had task forces on methodology about how to do appropriate analysis and comprehensive analysis of observational data One of our recommendations, which hasn’t been paid a lot of attention to,

and I think might be of particular interest to the FDA is that when you go into do an observational study, and that the insight is in a different population or it’s based off an insight of an RCT, to do a first analysis where you restrict the analysis to a population that looks like the population that was in the RCT, to see whether you reproduce the finding If you reproduce the finding and then you find something else, you would have greater confidence that what you found makes sense If you don’t find the same answer, then you have an opportunity to ask the question, what was different here compared to what went into the RCT? So it’s not even just about doing an analysis off a data set It’s actually understanding the insights in that data set And I think that Bill is absolutely correct Design is paramount And I think we need to find a way to bridge that anchor between the RCT world and in the observational research world – Again nodding, other ideas? – Emphatic agree (laughs) – Yeah, absolutely – Other ideas on bridging RCT to real-world evidence? – So I agree very much with your point I think it really help build that confidence with the real-world data I guess the other way that I would look at the bridging is the reverse bridging of what are the endpoints and data that are relevant in the real-world setting? And knowing that, for example, in oncology, resists is not being captured or used in the real-world setting or clinical decision So how can we bridge it back to clinical trials? So then the data that are relevant in the real world for treatment decision and clinical decision would be captured in the trial So then we have another way to bridge back – Great, great comments And Bob, I think you’re going to get the last question, comment for this round – [Bob] Actually, I wanted to ask about Alzheimer’s disease I presume the area of interest would be in something that delays its progression, or prevents its occurrence It couldn’t be just symptomatic improvement because we all know those effects are too small to be picked up this way That seems a potentially promising area, if one could define very well who the at-risk population is Presumably, before they’re overtly sick It’s not an easy thing to do But you can imagine endpoints that would be credible But they probably are two and three years away, so it sounds like a tough area, unless you’ve got a lot of time to wait – Endpoints, I think at least two or three years – [Bob] Endpoints would be an early diagnosis of mental dysfunction Or a next stage or early or something like that But all of those could take months to years Six months and a year But it seems sort promising, because those are things that are picked up in the system You know when someone has mental dysfunction – Is there a way to keep the? That’s going to have a longitudinal component, I think – Absolutely, yep – And is there a way to keep the, this one, Alzheimer’s did not make your early list of use case opportunities – We’ve certainly put a lot of interest in it – It certainly is an important one, so yeah, let’s talk about that for a minute – Yeah, I think to add a retrospective component when you have that is so important That is incredibly valuable I agree you get a longitudinal, but you can learn so much retrospectively And that is a full continuum – [Bob] That could also help you design a proper, a regular study, not a proper study, forgive me, sorry We don’t know that much about the natural history of people with early disease Maybe you could find out if people progress faster – Yeah, I think it requires us to be able to go back to look at the data and they was saying the data that have been in the system may not be adequate to look at that But certainly, I think this is an area that has a lot of potential and promise and would be of really critical for us to get that one – I think when you were talking about a national commitment, that’s where linking data sets, where you would have, kind of where the patients being diagnosed with Alzheimer’s and you’re able to go back into a course of history that is longitudinal and then digital biomarkers and a lot of the things we’ve been talking about with wearables and other things And what’s relevant to patients and all of that going back in time, I think one of the big challenges is where we’re seeing two and three years ago, the stuff you’re talking about isn’t necessarily linked to where they’re being diagnosed And I think that’s where that national commitment really comes in – [Bob] That really says that this hasn’t been talked about that much One of the things to do with this stuff is to get better natural history data than we now have Potentially as a control group, but also just to help you design the studies – Yeah – Absolutely – Well, I want to thank the panel You all covered a wide range of issues in getting to, as we talked about, a much better

and reliable evidence and a wide range of applications in getting there Ending up with Alzheimer’s seems like a really appropriate place to finish this panel So thank you all very much – Thank you (audience clapping) – We’re going to take a 15 minute break And our last major session starts at 2:45, thank you (people chatting distantly) Final session for the day

I want to thank everybody who’s stuck with us

for the whole day, been a lot of really outstanding

discussion and in getting ready for this last panel which is charting a path forward, talking with Jonathan, he said, “Okay, well now it’s time for people to start doing stuff.” So that’s what we’re going to talk about in this panel We want to discuss some of the ideas that came up today and also talk about some activities that are underway now, related to real-world evidence That will give us a chance to reflect on the discussion of the day To maybe think a little bit more about priorities And especially, Jonathan, practical steps forward that we can take from here And very importantly, we want to make sure that we’ve got a central patient focus in the work that we’re doing now So that’s going to be an important part of this panel as well So with that, I’d like to introduce our panelists At the far end, Greg Daniel, who you heard from earlier Deputy Director here at our Duke-Margolis Center for Health Policy Next to Greg is Jonathan Jarow, who’s Senior Medical Director at the Food and Drug Administration Who has been involved and done a lot of thoughtful work related to real-world evidence You may have seen his paper on this topic recently Joe Selby, is the Executive Director at the Patient-Centered Outcomes Research Institute And Preston Hinkle is a member of the Cystic Fibrosis Foundation’s Patient and Family Research Advisory Committee Preston, we’re really glad to have you here with us today, too So we’re going to do the panel in the same kind of structure that we had for our earlier panels, with some additional opening comments, thoughts from the group about some of the activities that they have underway already in these areas And feel free to add in or reflect on or push from what you’ve heard about during the course of the day today And if you don’t do it, I will, as we get to the discussion So Greg, please go ahead – Okay, thanks, Mark So I started out with some talking points this morning for today’s session And I’ve completely abandoned them and throughout the day, I’ve been jotting things down that I’ve head that gave me ideas So bear with me as I read from chicken scratch and I’m trying to figure out exactly what I wrote here But, in terms of high-priority topics that I heard either today or things that we outlined in our white paper as things that would be very helpful to start doing now Some of these things we’re already doing, but they need to be coordinated, are as follows One, improving standards in data collection methods So we know real-world data sources are very disparate They cover a wide range of types of data and different sources and there are challenges in making sure that when we’re using real-world data sources that we’re actually measuring what it is that we think that we’re measuring, and that we can interpret the data So a lot more need to go into standards and better data collection Another on is strengthening the methods for randomization in the clinical setting as well improving the credibility of observational studies And we heard a number of ways to do that, but more does need to come together And how we know that the observational study is credible? And how can we best do randomization in the clinical setting? What I also heard today is this idea of doing more common platform type studies Thinks that like PCORnet could support or other networks like that, where you can get a range of companies or a range of stakeholders interested in doing really high-quality real-world evidence generation for a number of products I also heard, and this is something I made up on my own, but I was listening and I’m forcing

it in here is basically what I’m trying to say here Evidence-based approaches for matching what we were talking about earlier, when we talked about research design And Bill, you were talking a lot about, it’s not necessarily the methods, but it’s the design of the research itself as a gap and linking clinical trials to real-world evidence generation But the bridge that I’m creating now is then how do you match up research design and all of the different flavors of that within real-world evidence with actual regulatory decisions? Simply, which types of regulatory decisions can be very well done observational studies support? Which types of regulatory decisions would you absolutely have to have randomization But as you get into the details of that, it’s not as simple as that There are a big range of things But we do need stakeholders to come together and really put a lot of thought into how you make those matches I did hear today engaging patients in a better way to learn about real-world evidence and real-world data But also to engage patients and have them participate with their data, but also participate in research design and communication of results Governance was a big thing we heard this morning That will continue to be an important emphasis to look at, both public and private sector ways to share data better and to build the infrastructure that we need to do more of the common platform type things And then novel analytic techniques that take better advantage of today’s computing power So every day, or maybe every week, maybe, brand new methods or more sophisticated methods for how to do real-world evidence development are out there and being tested and being further refined and developed We need a good place to continue to do that So all of these things can be done with demonstration projects But demonstration projects are great, but we do need them to be coordinated and transparent It’s unbelievable how many different groups and partnerships today that are all trying to do something in the area of improving real-world evidence development It’s going to be really hard to know exactly who’s doing what, what kind of projects are currently happening now, so that we can learn from those Understand where the gaps continue to be, is everybody focused on replicating clinical trials? Is nobody focused on some of the other things that we talked about? So it will take collaboration and coordination There are a couple of things coming up over the next year that I think are going to make a dent in some of that collaboration coordination I just talked about One of the things is the National Academies, their forum on drug discovery, development and translation They’re about ready to launch a three-part workshop series on how to do real-world evidence better, with a focus on methods and approaches across a range of medical technologies, not necessarily drugs But will be focused on methodologic approaches to better informed real-world evidence development And parallel to all of that, our center, the Duke-Margolis Center, will be launching a real-world evidence collaborative in October Where we’ve brought together a range of perspectives And we’ll be looking at supporting and broadening existing efforts So some of the things that Bill talked about in terms of what Optum is doing HealthCore is doing a lot, as we heard from Kevin earlier this morning And a lot of other groups are doing things that can be impactful and will be impactful And we’re looking within our collaborative to help learn from those lessons, to help identify where groups aren’t working and then to build on those So we’ll be conducting a series of workshops We’ll be producing a series of papers and identifying demonstration projects that can actually help fill in the gaps of where some folks aren’t necessarily working or focused on I think that’s everything on my chicken scratch I can read – Thanks, Jonathan – Well, I, too, did chicken scratch during the day So first of all, I would like to thanks the folks at Duke-Margolis for hosting this event and for putting on a wonderful meeting In addition, I’d like to thank all the panelists for the excellent conversations we had today And the people from the audience who participated, as well FDA is looking for stakeholder input on this subject It’s very important to us to hear from all of you in terms of where there are knowledge gaps that we need to help fulfill, either through pilot projects, guidance development, et cetera So real-world evidence is very exciting It’s a very sexy term, I’m surprised it’s not on the cover of Science, or maybe it’s not really science, so (laughs) it might not make it there But at any rate, it’s very exciting And I think the learning healthcare system is something that we all aspire towards

And I think various regions are ahead of us, in the US, in getting the interoperability and standardized data sets and stuff Sweden is someone that we aspire to achieve something like that or what happened in the Southwark Project is a great thing Because we all feel like we’re losing a lot of information And patients are on board with this I remember at a Friends of Cancer research meeting, Mark, I think you co-sponsored it or helped it along Patients were aghast that the information about their treatment wasn’t being used to help guide treatment for future patients We heard numbers today of three to 5% of cancer patients are enrolled in clinical trials So there’s a great deal of information lost and we hope that we can capture this information and use it, both for regulatory purposes, but also, there’s probably even a bigger interest amongst payers to make decisions about whether to pay for drugs and how much to pay for drugs I know in certain regions of the world, they again, market access initially, but are required to provide real-world evidence of the effects of the drug after it’s on the market in order to continue to get reimbursement And in addition, obviously clinical best practices would greatly gain from having all this information brought to bear Most of our best practices are based on expert opinion and not based on level one evidence Again, whether the real-world evidence through observational studies would be considered level one is another question So I want to say that I hope it’s demonstrated today and at other events and in our writings and speakings that FDA is in favor of this We don’t want to be viewed as a roadblock to utilization of real-world evidence in the regulatory setting And in fact, as I think has already been mentioned several times today, we’ve been doing this for a while Sentinel has been used for safety, primarily And we’ve approved a variety of drugs where demonstration of efficacy, I wouldn’t call it necessarily real-world evidence, but certainly based on observational findings and certainly with single-arm trials where we use external controls, which people call historical controls There is a great deal of experience already at FDA in doing this so I don’t think it should come as surprise to you And we already have companies that have approached us to discuss potential projects as well as a recent or soon to be approved drug where they used real-world evidence as I think one of the other panelists mentioned, one of the oncologists, to show that that population was not prognostically different from other sub groups of the same disease So I think that there’s certainly a role for real-world evidence We encourage people to pursue this I think that interactions with us, either through feedback on the white paper that was distributed prior to this meeting, or contacting the Office of Medical Policy through the mailbox that they have there with questions or complaints or ideas Probably no complaints, right? And in addition, meeting with FDA as you’re considering, at least industry folks, as you’re considering ways to incorporate real-world data and real-world evidence into the development program And as has been mentioned repetitively today, that could be used to just understand the disease better, used as historical controls, used in combination with a traditional clinical trial Or in place of traditional clinical trial I think it’s very clear that there is a lot of room for development and improvement in this area And I put it into three buckets The first one is data, and this has been said over and over again, but that’s the bucket of deplorables Is the data there? Is it reliable? Do we have data standards? Do we have a unique patient identifier to get? – Can somebody tweet that? (audience laughing) – Cross sectional and longitudinal And that’s one of the problems we have in the US Unlike Sweden, we do not have unique patient identifier Which would help, but if we were using social media or any of these things, pharmacy, pathology, how we do know that the, linking the patient to the drug that they’re receiving is very difficult in our current system

But enough said about data The second bucket is study design We had some comments recently about the importance of study design I think I’m glad to hear today that people repetitively said that real-world data and real-world evidence doesn’t mean you have to exclude randomization Obviously, FDA is very enthusiastic about randomization within the healthcare system There are other names for this, large simple trials, pragmatic trials I am disappointed and I think the onus is on us, that at every one of these meetings, people say, boy, I wish FDA would accept pragmatic clinical trials for regulatory action And Bob Temple and I sit there and go, “Why are they saying that? “Of course we would accept it.” We obviously have not been good at getting that message out If you do a randomized controlled trial, it’s a randomized controlled trial and we’ll look at it And this brings up just a little tangential point Definitions are important, we spent time defining what we call real-world data, what we call real-world evidence The bottom line is, whether you call it real-world evidence or not, FDA is going to review an application for a demonstration of efficacy based on the statutory requirement of establishing substantial evidence of effectiveness And we’ve historically used a P value of less than 0.05 in adequate, well-controlled trials Sometimes just one trial But that hasn’t changed Not the P value part, but the substantial evidence standard Actually the statute and the regulations do not mention P values So there are alternative approaches And we live in a very exciting time for regulatory science in that in addition to real-world evidence, which is in 21st Century Cures and FDARA, there is also this codicil about doing innovate trial design and looking at simulations and Bayesian approaches, sequential analysis So there’s a lot going on, both here, in the United States as well as in Europe And I know that the adaptive pathways, adaptive licensing pilot which is now closed, but they’re now working ADAPT SMART And they’re looking at incorporation of real-world evidence in the development of drugs for EMA regulatory actions So I think there’s a lot going on there But having said that, back to the bucket of study design There’s randomization and then not randomization I think everyone’s got it in their head that this is really all about, even though we want randomization, I’m not trying to discourage that All these discussions are really about doing a study design that doesn’t include assignment of treatment And that’s where you run into trouble Because establishing causal inference without randomization can be difficult And everyone says it’s based on effect size, but basically, it’s based on our confidence that whatever confounders, both known and unknown, that is exist do not wipe out the effect observed That’s really the bottom line Marc and ISPOR has done a wonderful job at trying to come up with a set of ways to do this that would create more confidence But again, the FDA has historically approved many drugs based on evidence that was anecdotal I think the smallest approval is Cholbam with 12 patients So we do have regulatory flexibility when it’s clear that the substantial evidence standard has been met The third aspect really falls in our lap, and that’s regulatory I think a lot of people need reassurance that good clinical practices can be followed And that FDA will accept data in review for the application I think that’s one of the uncertainties that companies have about embarking on this area Our current guidance on this, ICH E6 R2, really doesn’t deal with these non-traditional trials And so it’s incumbent on us to revise that guidance And that work is already happening to do that, but I would not expect that within the timeframe, of the framework that we have here for real-world evidence So I want to just, some final points

FDA continues to support Sentinel It is exploring ways to use that for things beyond just safety analysis Again, we’re open to feedback on all of what happened today, the white paper, other issues that come up, and that can be either done by sending something to the mailbox or a meeting with us Thank you – Thanks, Jonathan Just to be clear, that mailbox is not like the mailbox outside of the FDA office – Right – It’s an online – Silver Spring, yeah – It’s an online place – It’s CDEROMP, one word, – [Mark] It was on Jacqueline’s slide earlier – Okay, it’s on her slide – Joe? – Okay, thanks, Mark And Jonathan, there was one group that you left off your thank you list and maybe you felt you couldn’t, but I think the FDA should be thanked very heartily It was you that called for the meeting to happen It’s your presence here, I think, in large numbers that contributes to the rest of us being here and to the energy in the room So we’re very appreciative of you having convened us all I want to start with a little comment about progress I got here about six years ago as PCORI was getting off the ground And the first meetings I went to about real-world evidence were more along the lines of, is there any possible use for it or no? Is the fact that doctors are entering this data and they got it wrong the last time I went to the doctor’s office and that proves that it’s worth nothing for any purpose We’re not hearing that anymore And we’re actually now hearing just an amazing array from a lot of different corners of interest from natural history, the use of it for natural history studies and improvement and enhancement on natural history gathered in many other ways, to predictive modeling, which we at PCORI are very excited about, to machine learning So all of those I think are, we have thought about them at PCORI We’ve thought about them at PCORnet and appreciate that really, there are aspects of precision medicine, there are aspects of getting the right treatment to the right person So very exciting to see that so many people in this room have an angle on that that points in the same direction So we talked about, I think this is next steps or the way forward or something like that So I have five things I’d like to just very briefly mention I have to say a few more words on linkage Not that it hasn’t been talked about a lot already Talk about societal change, something that Kevin brought up and I mentioned earlier The power of demonstration projects Standardization, trial infrastructure Those are the topics So first, with respect to linkage It hasn’t been quite said today that you really can’t do anything in observational analyses if you don’t have a denominator like the denominator provided by claims data, whether it’s a commercial plan, Medicaid or Medicare So all the data that EMRs gather is on an unspecified cohort of people You don’t know how many people would have used that system, but just didn’t You don’t know how many people used that system for part of their care, but had a diagnosis or an outcome elsewhere So we think that for either observational studies or really for efficient large trials, linkage is impossible and PCORnet has that challenge because we started with EMR data and learned over time how tough the linkage issue is We’re currently co-funding with the FDA several small projects on linkage, both in devices and in drugs to be And particularly looking at possible linkages between Sentinel, a claims database with well over 100 million Americans in it and PCORnet, an EMR based system with well over 100 million Americans it Probably not the same 100 million, but substantial overlap It’s very interesting and just a last word on linkage I think, is that the thinking behind PCORnet is it isn’t going to work is the holders of the data, from the patients to the clinicians to the systems to the payers, if they aren’t involved And it’s quite interesting that the linkage problems are really between the delivery systems and the payers Now that’s an interesting pair, the delivery systems and the payers As somebody said, they do business together this morning and that’s a challenge that’s part of the barrier But can you imagine the relevance of the questions to which the systems and the payers agreed this is a relevant question? This is a question worth contributing our data and sharing it with the other side, so to speak So this notion of bringing maybe unlikely partners together around data is just critical

It’s very interesting to me that it’s come down to that Connected to the question of linkage is this question of society and particularly the role of patients And today has moved me, I think I have long subscribed to the notion that it’s unethical not to do the most you can with the data that are being generated in delivery of care, to learn how to do that better Ruth Faden and Nancy Kass’s argument from several years ago, I’ve always subscribed to that But I think, and I took note that Jason from National Health Council said that they had just convened a group to talk about real-world evidence I took note of what Sally and Kevin and others said in that first panel today And I think I can say that as one way forward, we would like to convene or co-convene or join in a serious meeting on this question is it time for patients, and the community, patients, people, to begin becoming more aware of the fact that these barriers to linkage, barriers to the use of data are getting in the way of improving care and to have their voices heard I think we need to have IRBs there We need to have institutions there, even more I honestly don’t think it’s primarily a HIPAA or IRB problem So linkage, so societal change, this ultimately does lead to the learning health system, I just want to be on record as saying that I think a learning health system naturally leads to the prospect of hosting everything up to and including pragmatic clinical trials with randomization I think that once you get into it, you’re stuck, you begin understanding that research has value And then you begin understanding that some research requires randomization and there you have it So I don’t think there’s a disconnect The power of demonstration projects So Rich has left the room, Rich Platt, so I can say that most of anything I know about big data I learned from Rich And one of the things he always told us on PCORnet and before that with Sentinel, was that you learn most by getting started doing the research So you don’t actually learn it by sitting down in a group and trying to imagine all the problems you’re going to run into You can find them in a heartbeat by starting a study So in PCORnet, we have several demonstration projects One is a very large study of bariatric surgery and this joins data from across 60-some-thousand, I think it is, maybe 100,000 But I believe it’s in the 60,000 range of bariatric surgery, including very large numbers even in adolescence So out of that, the researchers came up with a remarkable document that was a series of learnings And they were about the issues you expect about the degree of missingness and about the importance of the degree of missingness And the need to standardize data across systems and how they’re not So I truly think that ADAPTABLE is the trial that Adrian Hernandez talked about a little bit earlier today And in ADAPTABLE, we’re learning about, among other things, e-recruiting So most of the patients in ADAPTABLE have been recruited through the portal of their health plan, wherever that is So we’re going to continue funding demonstration projects We have number of them and as I said, a couple of them in collaboration with the FDA But we’d love to see others join us, so we really invite you to get in there and projects that have a piece of research, demonstration projects have a legitimate research question, but you realize that along the way, you’re going to be blazing new trails You’re going to be standardizing new data That takes me to standardization of data And we are supportive and we fund data standardization The common data model that either OMOP or Sentinel or PCORnet have, any of those, is still relatively close to what it looked like when there was just claims data It started with claims data And we’re only very timidly, because it’s so complicated, going into the world of EMR data We need to bring people together, clinicians and researchers and holders of data To talk about measures that are not in the common data model yet Pathophysiologic measures like lung function and heart function, ejection fraction Just to name a couple More sophisticated, more complex laboratory measures, genetic information on tumors and journaling as well Data standardization is an area that I think both takes the real-world data to a new level to where it’s really useful But it also spawns research in the process I have a meeting coming up with the Oncology Center of Excellence with Amy from Flatiron

with ASCO and a number of others to begin talking cancer by cancer, about what are the data elements in real-world data that you really need to be able to do either large observational studies, say you have off-label prescribing, or pragmatic trials, outcomes trials So I just encourage that illness by illness And ultimately, I just ask the question of whether you don’t agree that ultimately, the data that we need for pre-approval trial should be the data and the outcomes we need for post-approval trial ought to be the data that clinicians and patients need for clinical care I think, ultimately, if you want to be patient-centric, that’s the kind of outcomes that you would need to standardize and include Trial infrastructure, this is I think the very last one ADAPTABLE taught us that it’s one thing to want to do a clinical trial in a real-world setting, you know, those of you who do regulatory trials know that you have a vast infrastructure to do them And we are trying to conduct trials that don’t intrude, or hardly intrude, on clinical care But we don’t really truly have the infrastructure there And to do trials of serious questions like does changing a medication make a difference? We are going to have to invent a trial infrastructure PCORnet and the Patient Center Research Foundation, which is now the not-for-profit that PCORnet stood up to help it with sustainability, will be working on this Will be working on beginning to strengthen the trial infrastructure in real-world settings for pragmatic trials So those are some areas that PCORI and PCORnet will be involved in coming up and with that, I’ll stop – [Mark] Great thought and Preston, really glad that you’re with us this afternoon, too – Thanks, well, good afternoon I am here today as a patient representative There’s been a lot of conversation today about the patient’s perspective and I get to be that voice, no pressure I was diagnosed with cystic fibrosis at the age of three For those who don’t know, CF is a genetic disorder that affects the regulation of ions and water in the body It causes bodily secretions like mucus to be very thick and sticky That has a host of complications, including in the lungs where particles, bacteria, can get trapped, cause infection Having a chronic illness like CF is incredibly formative A patient has a perspective that is really unique, from that of a researcher or a doctor A researcher might approach a new treatment as what’s the data? What does the data show? A doctor would say, what are going to be the outcomes for my patients? As a patient myself, I care about these things, but I have to think of other considerations, such as, how does this treatment make me feel? Or even, do I have time in the day to add another treatment? It’s really going to be important, I think, going forward with the real-world data, real-world evidence conversation to include the patient perspective To bring that voice in The Cystic Fibrosis Foundation has been an incredible pioneer in this regard I’d like to highlight a couple of their initiatives For over 25 years, the CF Foundation has run their patient registry This covers almost 30,000 individuals that which consists of about 84% of the CF patients in the United States They collect data at every visit, including health metrics, demographic information, bacterial cultures It’s a huge list of data and it’s just a really incredible wealth of information There’s so much there that the CF Foundation literally did not know where to start The solution that they came up to that with I think was really incredible They said, let’s turn this over to the patients To the caregivers, to the people that research affects the most They put together a team called The Patient and Family Research Advisory Committee, which I was able to serve on It consisted of patient and patient family members And we helped to launch the Insight CF Project This was focused on collecting questions from patients, family members, healthcare providers, anyone with a personal stake in cystic fibrosis research

And simply saying, what are your questions? What can we help to answer? Here’s the information that we have with the registry, what of this do you want to know more about? So we had a really incredible response, over 400 questions submitted Of those, about 150 were feasible We were able to narrow that down, resubmit it to voting and came out with three amazing questions that were sent to a research team And the rest were held onto, because they did give a really incredible insight into the current state of CF patients, their minds, their concerns So this, as a patient myself, who was able to submit a question, was intensely gratifying The ability to submit my own concerns and know that they were being listened to It was just, it was a unique experience, it was incredible So I think that this should be a role model for other organizations moving forward Connecting directly with your patients Listening to what they want, what they need Opening up the discussion to them One other initiative that I wanted to highlight of cystic fibrosis in the discussion of real-world evidence is the label expansion in this past summer of a drug called Kalydeco This is from a new class of drugs called CFTR modulators, which actually go in and assist the protein that’s malfunctioning in the cells to achieve some function It’s baby steps toward a cure, it’s really shown incredible results, but it’s limited by genotype So the populations are difficult to do traditional randomized clinical trials with Over the summer, the label expansion was able to be done using in vitro data, which was backed up with a safety profile that was garnered from sources like the patient registry and other sources of data And so it was, I can’t speak to the methodology, I’m not an expert there But as a patient, this was phenomenal news I personally am not eligible for one of those drugs yet, so to me, as a personal stake, expanding those fields, expanding the breadth that these drugs can cover should be one of the most primary goals of CF care in the near future This is yet another example of how real-world evidence can really make a difference in the lives of patients To sum up, I just would like to encourage everyone, keep the patient’s perspective in mind A patient is an expert in their own experience Use that, utilize that, and I think that by doing so it’s going to drive some really incredible results going forward – All right, thank you very much, Preston I guess the one thing, the one reason we thought it was very important to have your perspective on this panel is, I know you’re not a methodologist, but as many of you’ve heard today, the issues that stand in the way of real progress on real-world evidence are not, they certainly involve methodology, but they also involve more basic things Like going back to Joe’s list and some of the points that you all raised around patient support for the research Around being able to capture longitudinal data Around being able to capture meaningful information on outcomes or questions of interest And then formulate those questions in a way that everybody can get behind Payers that might contribute data, researchers, clinicians and of course the patients It seems like a lot of the issues that we’ve raised here today have been addressed in the CF platform that you’ve been an intrinsic part of developing, not to mention contributing to And you did talk about some of the implications and hopefully lessons for other areas And we talked about other potential applications today, too Anything else to add on this point? I’d like to ask the rest of the group about how a model like this could be applied, or what are the barriers to applying it in these other areas, where we’ve got lots of use case examples that we talked about today Where there is some level of patient engagement, but not maybe to the extent that we’ve seen with the CF work that you’ve described So first, any additional comments you’d like to make on this topic and then I’d like to ask the rest of the panel about how much of a model can this be

for real-world evidence progress in other use case areas? – Sure, so one other thing I can highlight, I guess, is specifically in the context of real-world data for real-world experience, is the questions that we received through that project that I was a part of were incredibly varied and I was just blown away by the amount that were actually answerable with that registry And to answer some of these questions, using a traditional randomized clinical trial would be a significant challenge, just out of sheer numbers However, with initiatives– – So many of these genetic subtypes, there are just a few patients, even though you found most of them in this effort, yeah – Sure, and so, in that registry, it’s really incredible because almost 30,000 patients That data is there, that data, we don’t have to recruit people, to design a study for, we can just go to that and look And the implications of that to me are incredibly profound – Greg and then I know Joe has got something to say – Just one piece of that that I want to comment on is that Preston seems pretty motivated to participate in that data, just by listening to you, I could tell that knowing that your voice was being listened to or that something positive was happening with your data was enough to motivate you to continue to participate When we talk about real-world evidence, we often talk about claims and electronic health record data And while there is missingness within that data, it’s pretty much there If you’re a provider or hospital, you need to submit the claim to get paid So that’s going to be there If you’re a provider that use an electronic health record, by and large, you’re documenting data into that health record I don’t know if it’s accurate, we don’t really know, nobody really knows, but it’s there So now we’re talking about unique data sources from patients And there are ample ways that, in technologies that we have where patients can contribute data through the registries like the CF Foundation or even getting into mHealth technologies and apps and things like that But I think we need to think about that fundamental element that I picked up on from Preston, is that it will depend on how motivated the patients are to continue to use those technologies, or continue to, they don’t need to do that to get paid like providers when they’re submitting claims And they don’t need to document it to treat patients like providers and payers all do those things What is going to motivate patients to contribute to those studies, to keep using those wearable apps that are contributing to research? And something simple as letting those patients know or having them experience the fact that their data are being used in a meaningful way can go a long way And I just would encourage groups that are looking into how can we better use patient datas to think about how contributing data can actually have an impact on those individual patients that are participating – So I just want to say first of all, it’s not a coincidence or just a drawing a ticket out of a hat, that Preston wound up here today That everybody thinks of the Cystic Fibrosis Foundation as the paradigm and some happy coincidence of very broad coverage, there was one really national cystic fibrosis patient organization, one Number two, to my knowledge, you really linked the foundation to drug development efforts in a very tight way and I think, number three, you were a rare condition with no good treatments, and so you’re highly motivated So a lot of things happened right, plus it must have been just unprecedented leadership, just incredible leadership, but there are a lot of patient groups and we know a lot I’m sure the Patient Powered Research Networks in PCORnet who aspire to be like your foundation So I think there could be ways that we learned – [Mark] Are you seeing some promising developments in that direction? I know much of the PCORI work has helped support – We’ve just issued an announcement that actually is directed only to the PPRNs So the big CDRNs can’t apply for it – [Mark] Maybe just back up to explain the PPRNs – Okay, good, sorry Sometimes I forget So PCORI funded PCORnet, which is a very large investment And it’s a network of networks and in PCORnet, there are 13 very large clinical data research networks, which are EMR driven Every one of them has several millions of patients You add the 13 up and they’ve got somewhere over 100 million persons with records in the database

So that’s the CDRNs, and then there are the Patient Powered Research Networks and these are groups often with an advocacy organization involved Sometimes with an academic base but, of patients with a single or group of very closely related conditions like vasculitides, for example Epilepsies for another And they aspire to become more involved in research Some of them already are, some of them have partnerships with industry sponsors already Maybe especially the rare disease ones But we’ve just put out an announcement that will bring the PPRNs together with another funder, they must bring another funder, so industry sponsors, or other foundation sponsors Advocacy groups could join in funding of it So we are doing what we can to try to make these groups, help these groups get to the point where they’re more familiar with, more comfortable with participating in trials research And they’re very good at collecting data Transferring that into analyses that lead to cures, or I’m very interested in the questions you’re putting in your pocket You know, the ones you didn’t study yet at PCORI I think PCORI might have a real interest in what those are – Sure, I’m interested as well (audience laughing) – So I think there are several lessons that can be learned from what the CF Foundation has done Some related to drug development, some actually independent of drug development I think the CF Foundation had a major impact on survival of patients with CF independent of drug therapies And through their identification of best practices and how to manage patients with CF But apropos drug development, I think the patient registry is a useful tool One, for learning about natural history We keep talking about external controls For two, identifying subjects for enrollment in clinical trials and then three, for a collection of real-world data and real-world evidence for primarily, like in this case, for label extension Whether it’s in rare subsets or after accelerated approval, the confirmed benefit So in a phase IV environment, such as a post-marking requirement So I think that that information is very useful and cystic fibrosis has been good In a way, a platform trial or a platform is very analogous to a registry And our folks in the medical device arena are way ahead of us You’ve alluded to gastric bypass or bariatric surgery And they’ve been dealing with this for a long time There are a lot of lessons that we could learn from our colleagues at CDRH, just for, maybe not fortuitous, but for the circumstance that most devices are approved outside the US and are used for many years before they get on the market in the US And therefore, there’s a wealth of experience in clinical practice So they use registries very frequently for demonstration of efficacy of devices So I think there’s a lot we can learn from these lessons And it’s definitely important I think cystic fibrosis has certainly served as a prototype of how we could do this in a variety of diseases – All right, you all have heard a lot of really insightful remarks from this panel about next steps from what we’ve heard talked about today Really some interesting ideas about extending this patient-focused platform into other areas of real-world evidence development Before we wrap up, this is a chance for final questions, comments, for the panelists about any issues that you’ve heard about today that you’d like to follow up on Any ideas, thoughts about next steps So I’d like to open this up to questions and comments And again, those of you who are just joining us online, feel free to email us about your question, too – [Michael] Michael Liebman, IPQ Analytics Joe, you mentioned bringing in some of the parameters from things like EHRs and you specifically mentioned things like ejection fraction And bringing together clinicians and so on I think when you start to look at some of these parameters, one of the things you need to look at is that clinicians will deposit parameters or measure parameters based on guidelines and procedures that they’re trained in and used to applying Especially if we’re trying to drive the integration or the merger of research and clinical practice, we need to look at the fact that things like the echocardiogram

or the ejection fraction as measured have a very large number of additional parameters that never appear in EHR and that, if you’re looking at a cardiac patient, a cardiac physiologist has a completely different perspective on what’s important about heart function than a cardiologist So it’s just an opportunity as you’re planning, as you said, to make sure that other perspectives are able to be incorporated in that early planning stage – Thanks – I think there’s an enormous amount of work that needs to be done with EHR to make it fit-for-purpose And we’ve talked about it all day and there have been meetings, all-day meetings, just focusing on that one issue I think one thing to point out, sort of a tangent about what you said, but related to patient-focused drug development, is that the endpoints that we use right now So HAM-D scores for depression, pain scores for treatment of pain SF whatever, 26 or whatever the number is for But all these things are not used in standard clinical practice and are not there When you start working to change the EHR, if you can actually change the EHR in the entire country, to a uniform data set, as you do that, you have to keep in mind that we may be changing our endpoints To, someone said earlier, whether they could be home or at graduation or whatever it was But it’s different from what we’re using in standard clinical trials We have to keep that in mind It’s a lot easier to adjust a CRF for an individual trial to say, okay, this is going to be our primary endpoint How someone feels about fatigue is what we’ve learned is the most important symptom of this disease for these patients with this disease We’ve been using some other measure and then all of the sudden, that’s not in the EHR And this is most of what we’re talking about doing, at least for observational studies, is probably going to be retrospective So it’s not going to be there It’ll be a missing element So it’s just something to consider as we roll up our sleeves and start doing all the work that’s needed to be done to get the EHR fit-for-purpose for real-world evidence generation – Thank you, other questions? It looked like – Yes, hi Jennifer Christian, I’m Vice President of the Clinical Evidence at QuintilesIMS and I have three quick comments The first is my group works a lot on innovative studies, pragmatic trials, those that combine prospective data collection with existing databases As well as extension studies, so those that roll over patients from RCTs and follow them for long-term benefits and risk And one of the big questions we get, as we’re working with pharmaceutical companies are what evidence do you have that this design has been used before for regulatory decision-making? A lot of case studies were used today, but I would encourage one of the outputs from this to be a way of sharing these more systematically and describing some of the context that went into the reason both for using it and the impact that it had The second comment I have is around endpoints And I love Joe’s comment that they need to be really what’s happening in practice So I would just give an example in working in Crohn’s disease, the traditional endpoint that’s often used to measure remission is the Crohn’s Disease Activity Index, or CDAI, which requires seven days of a patient diary prior to coming in, a number of biomarkers to measure, and then the point at which the clinical evaluation is made And if you ask gastroenterologists, they say, we don’t use this in practice Could we use something that’s much simpler? Maybe a five or six question tool like the Harvey-Bradshaw index? In which case, maybe we could use a pragmatic study design then Or look at reduction in surgeries or hospitalization, in which case we could use claims data So a big a-ha I think in the planning of these is what endpoints can we use that then drive decisions around the design? And then my final point is just to plug, really, pre-specification has been brought up, and I saw InSet being mentioned, there is ROPER, the registry of patient registries where this could be listed And we have been with HRQ on a new ebook that’s coming out this fall called

21st Century Patient Registries Which, Preston, it does include working with patients throughout the study design and conduct of the study as well as designing more patient-centric studies and using digital technologies, so look for that – Great, thank you very much for the comments I had a question over here and then over there, too Where’s the microphone at? Yeah, go ahead – [Lisa] Lisa Goldstein with the American College of Cardiology, I have two questions One, I’ve heard a lot today about how patients want to contribute their data and want to participate And I don’t necessarily disagree with that, but one of the things that we require in this country is patient consent We keep talking about putting together different data sets and existing data sets for different purposes, but we’ve yet to solve the problem of consent So how do we propose to address this issue of consent for ongoing future use while ensuring that patients remain engaged and aware of how their data is being used? So that’s the first question And then the second question is, a lot of what we’ve talked about today seems like it would fit purposes for academic medicine, but not necessarily those in private practice Yes, they have electronic health records that they hate, where they feel like they’re clicking different, constantly clicking all these different data elements and don’t understand why they need to do that So how are we going to keep it to a manageable amount of number of clicks so that they can still engage with their patients properly? And provide appropriate care while ensuring that we’re not developing a framework that only works for academic medicine Injecting an entirely different bias into that data set? – Great questions, and we have to keep the answers short We did talk about informed consent approaches a bit earlier today, but any brief thoughts? – Yeah, I’ll skip informed consent because I’m not a lawyer In terms of the last thing, you’re absolutely correct We have to decrease the burden to physicians in the EHR, so by modifying the EHR, we don’t want to make it worse What we want to do, though, it is make it fit, if you make it fit for clinical practice, it’s going to be fit for research, in theory So they will then like it Right now, they don’t like it, because right now, it’s a tool primarily used for reimbursement And it’s one that they do begrudgingly because that’s how they get paid I think again a lesson learned, and I’m not sure that this may be transferrable to the US, is the Southwark project And they found that in having pharmacists and physicians enlisted and that they knew they were participating in the research, this was real-world evidence based on one drug, but for two big studies One on COPD and the other on asthma They found that they had greater buy in So they were enthusiastic to participate and then the side benefit potentially, is greater adherence One of the problem we have here in the US, like with atrial fibrillation as an example, there are many patients who should be prescribed anticoagulants who aren’t And then once they’re prescribed, they don’t necessarily keep taking it So there’s hope, I don’t know if there’s a definite it will happen, but as physicians participate in these studies and know that the evidence was generated from their patients and their practice, maybe there will be better uptake of good clinical practices – Joe? – In response, I’ll take the first part about the IRB So we’re here talking about real-world evidence and I really appreciate the fact that the FDA has said several times that real-world evidence often comes from randomization But sometimes it doesn’t And in the case of registries, it typically doesn’t So I would say that in terms of HIPAA and the constraints on the use of the data for research purposes that could not otherwise be done, I think that the use as well, arguably perhaps, linkage is covered by HIPAA But we have not had the conversation with the public about this, so I think there will be people who raise the issue that you just raised And that’s part of this societal change – That’s what you were just talking about, some next steps on that – We simply haven’t ever had that conversation, coming from Kaiser, places that use the data a lot didn’t really talk about it a lot until the publication came out and then it was a good time to celebrate – So great questions, we are about out of time So I think we have time for maybe two more quick ones There’s one there, one here and then one there – Yes, thanks – Sorry, over here, first, – [Solomon] Hi, so my name is Solomon Iyasu, I’m with Merck, great discussion I wanted to have some maybe commentary from the panel on what I see as the fundamental

paradox in our evidence generation in the lifecycle of a product As you know, drugs are approved based on the narrow indication clinical endpoint that is validated and for a short time There are a lot of limitations and they are approved for their purported use and benefit and under specified conditions of use Which is very different from what happen in real life But as we grow into the post-approval phase we’re learning more and more about the safety profile of a drug Both labeled and unlabeled use under conditions of use approved and unapproved So there’s a formation asymmetry in the evidence generation landscape that builds up over time where we’re getting more information about safety which we have a certain amount of comfort Although they are coming from real-world data, but we are not developing the same type of information to capture the full range of benefitw that probably wouldn’t– – Including effectiveness and so forth – [Solomon] So what should it look like in the evidence generation landscape to really characterize and capture all the safety and all the benefit profile apart from what is measured in clinical trials to really define the benefits of a drug as we go forward that’s acceptable in a regulatory sense – That’s the dream of real-world evidence Is at the time of approval of any drug, there remain literally dozens of unanswered questions We sometimes don’t even know what the right dose is Based on the development program Potential expansion or limitation of the indications Bob’s upset that I’m showing our dirty laundry in public – It’s so true – So there’s so much more we could be learning about a drug So we have an option, one is FDA could say, well, we won’t accept any information except if it’s a randomized controlled trial for these literally dozens of questions And so therefore, they don’t get done So these questions remain unanswered Or we could say we’ll consider other sources of evidence, observational, whatever, to inform labeling, clinical practice, et cetera – And new kinds of randomization in this real-world pragmatic context, too – Or not – Or not – [Sheila] Hi, Sheila Weiss from Evidera And it’s been a great day and I thank you, this panel and the others And again reiterating, FDA uses real-world evidence every day and they’ve used it for a long time, particularly heavily on safety But what I’m hearing here is kind of a broad sweep, where are we going to place this type of data in regulatory areas where it hasn’t been used? For example, drug approval, and we know that definitely within orphan drugs, that is happening more and more But where else can it go and how far can it go down the line? And I think what we haven’t discussed is where something, a question, is regulatory, for something that is being submitted to a regulatory agency for a key decision And when is something that’s a public health issue, where we’re looking at comparing drugs or the usefulness or the side effects of drugs and how they’re valued by the patients among all the treatments that are available Those are two very different and both very important questions I see Marc Berger shaking his head there But I think we need to think about it that way, because the issue of approval is what is the quality of the evidence? And sometimes we don’t know the quality of the evidence until we get the answer from the study Because obviously, the larger the difference, the more we are willing, in most cases, to allow the rigor to be less than if it’s a small difference So I’d like to hear the panel’s take on that – And again, it does come back to some of the challenging core issues that we’ve talked about today So any final, quick comments on this question? Or any final comments on the panels? This is your last chance – You raised several questions And the one about when do we believe an effect, direction and magnitude, based on observational data versus randomized controlled trial is a thorny one for us If you saw it, we use it all the time for safety We’re not required by statute to have substantial evidence of a safety finding And we struggle, even with the Sentinel database

and the 190 plus million people in it We struggle with safety signals, behind closed doors, scratching our head, trying to figure out is this real? Is it not real? Should we put it in the labeling? Should we not put it in the labeling? It is very difficult when you don’t have randomization Where you can establish causal inference Because that takes care of both the known and unknown confounders And I just want to say that that’s going to be a very difficult thing for you all and for us going forward And that’s why we keep on stressing, you can do randomization within the healthcare system – And appreciate the collaborative approach to try to address these issues of observational data Any final thoughts from the rest of the panel? – Just following up on Jonathan’s, I think we should reconvene in two years and ask that question, have we gotten real-world data and the methods and the approaches and the capacity to randomize in place sufficiently that we can now begin talking about using real-world settings for regulatory trials – Including on effectiveness, yes – On effectiveness, yes – Other final thoughts? – Yeah, I would just say on this last point, and it was brought up before, this question about reproducibility Is it of the methods or of the results, and I think it’s both You shouldn’t be doing research unless your methods aren’t transparent enough that somebody else can actually do the research So you have to be transparent with the methods that are being used But when it comes to the observational studies, that we’ve been talking about today, consistency of results in multiple studies So as Marc, the example of four different studies in completely different data environments, that you’re seeing consistent results from study to study, that means something and that had some value So I think a good step forward is trying to get all of you, our thought leaders, together to think about, okay, what are the appropriate ways to make sure that our observational studies are credible? Consistency of results, reproducibility of the methods, pre-registering a priori the protocol and the analytic plans and sticking to those are examples, but we need more information on what the best approach is – Preston, last word? – Last word, big pressure With regard to this specific last question, as I said, I am not an expert on methodology Again, I just have to reiterate that turning it over to patients and having the patient-centered approach is the best policy here Keeping in mind that whatever we do, it should protect the privacy of patients It should protect their safety And it should be really geared into the mindset of what is going to make their lives day-to-day a better place? – Yeah, answering the questions that they want answered Thank you all very much (audience clapping) (Mark drowned out by clapping) For closing, today has been one step in what, as you’ve all seen, is a big ongoing journey to getting answers to the kinds of questions that Preston was talking about in terms of using and leveraging real-world evidence There clearly is a new level of interest, new capacities and hopefully coming out of all this, hopefully within less than two years, Joe, some concrete steps that we’ll able to see as progress and impact from the kind of work that’s been done today But I’m going to turn this over to Greg for thanks and next steps So that suffice to say, we really appreciate all that you’ve all have contributed to this effort And in advance, for the next steps in making progress on real-world evidence – Okay, I’ll do that right now, so just stay here and I’ll just do it from my seat I actually didn’t know that I was doing closing remarks So this is the first time that I’m reading these talking points You’re here, why don’t you do it? (audience laughing) Anyway, I want to thank everybody today for participating, the 1,500 people online and I’m sure due to our tweeting, we probably got up to 1,600 of the folks dialing in for the webcast, thank you for doing that I do want to mention that we are going to have an informal comment period So starting from today and then 30 days from now, so up until October 11th, we will be accepting comments through email to all of you to that email address that’s on the screen We’ll do our best after that 30-day period to synthesize that information and put out something in conjunction, or an addendum to the white paper, summarizing some of the major comments that we heard, so I encourage you all to submit your feedback and comments to our white paper through that route I’d like to specially thank the FDA,

not just for helping us with this particular meeting, but we’ve been working with you very closely over the last, two years or so, when we started talking about this topic And we’ve been very appreciative to all of the comments and feedback and participation in the activities resulting in the white paper and today’s meeting I’d particularly like to thank Jacqueline Corrigan-Curay, Dianne Paraoan, Melissa Robbs and Kayla Gavin for a tremendous amount of work and you (laughs) for all of the work that you’ve done, Jonathan, Rich Moscicki, Jonathan Jarow, who’s sitting right next to me, representing FDA For the Duke-Margolis staff, Morgan Romine, who’s done a tremendous job working with all of the FDA folks on our white paper and making sure that we’re covering all of the bases as well as designing today’s event, which was a huge success Katherine Frank, Ellen de Graffenreid, Elizabeth Murphy and Sarah Supsiri So thanks to all of your participation, please provide us your feedbacks And have a great rest of your afternoon (audience clapping)