What is a margin of error? This statistical tool can help you understand vaccine trials and political polling
Ofer Harel, Professor of Statistics, University of Connecticut
Posted:
Updated:
Billion Photos // Shutterstock
ws of vaccine trials from companies such as Pfizer, Moderna, and AstraZeneca.
Already there have been some promising developments. On Nov. 16, Moderna released the results of a study that showed their vaccine has a 94% rate of effectiveness. While the results of their 30,000-person Phase 3 trial have not yet been published, researchers said that the results show that the vaccine is safe and effective. On Nov. 18, Pfizer and its partner BioNTech announced it concluded its Phase 3 study (which began July 27) and determined their vaccine is 95% effective. Pfizer and BioNTech will submit a request for Emergency Use Authorization by the FDA as soon as possible and plans to share their data with global regulatory agencies.
The news from both companies has given people hope that the SARS-CoV-2 virus that causes COVID-19 can eventually be controlled. However, there is also mistrust surrounding vaccines, and often a lack of understanding about how they are created, how they are tested, and how safe they are. If people don’t trust the vaccine, then people won’t take it, and the pandemic could go on longer.
Some of this lack of trust comes from a lack of information or misinformation. In order to demystify vaccines and the vaccine manufacturing and approval process in the United States, Stacker consulted the Food & Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), and other public health sources to better understand and explain vaccines to the general public.
After going through these authoritative health information sources, Stacker identified some key terms to help readers better understand the types of vaccines and how they work, and then listed the many steps involved in the creation, approval, and distribution of new vaccines in the U.S. We then created a list of 30 key terms and steps, showing how intensive and precise the vaccine creation and approval process is, and ultimately that vaccines are safe and effective tools for fighting disease.
The U.S. Food and Drug Administration (FDA) protects public health by ensuring the efficacy and safety of biological products, including food and drugs such as vaccines. This means that for any vaccine to be approved for distribution to the American public, it must first be approved by the FDA.
The specific area of the FDA that deals with vaccines is the Center for Biologics Evaluation Research (CBER). This center regulates biological products using an array of regulatory measures, such as the Public Health Service Act and the Food Drug and Cosmetic Act.
The process to create a vaccine is historically a long one. It begins with scientists and researchers identifying the antigen, which is the part of a germ that the immune system can recognize and attack to prevent the targeted disease.
Vaccines work by stimulating a reaction from the immune system. They do this by essentially tricking the body into thinking there’s an infection. While the body may occasionally experience minor symptoms of infection after getting a vaccine, the kind of pseudo-infection introduced by the vaccine almost never causes illness.
Whole-pathogen vaccines are the traditional type of vaccine. These vaccines contain entire pathogens that have either been killed or weakened enough that they cannot cause disease. Because they have whole pathogens, they elicit strong immune responses. However, not every disease can be targeted with this type of vaccine.
Unlike a whole pathogen vaccine, a subunit vaccine uses just the antigens to best stimulate the immune system. This vaccine design is safer and easier to produce, but it often requires the addition of adjuvants, components that elicit a stronger immune response, because the antigens are not sufficient on their own for long-term immunity.
Nucleic acid vaccines use genetic material to encode the antigen or antigens needed to produce an immune response from the body. This allows the body’s own cells to produce the antigen(s) using the genetic material. The advantages to this type of vaccine are long-term immune responses, scalability, and vaccine stability. Some of these vaccines are based on mRNA (messenger DNA). Both the Pfizer/BioNTech and Moderna coronavirus vaccines use mRNA.
Some vaccines require more than one dose. There are a few reasons for this. Some vaccines do not provide much immunity in the first dose, and therefore need more. In others, immunity wears off after time, and “booster” doses are needed. In some live vaccines, multiple doses make it more effective. And in the case of the flu vaccine, a new dose is needed every year because the flu virus that causes the disease varies year to year.
Before beginning an Investigational New Drug (IND) application, a vaccine must be screened for potential danger to animals. These take place through animal pharmacology and toxicology studies, taking preclinical data to allow an assessment as to whether the product is safe enough to begin testing in humans.
Once the screening for potential danger to animals is completed satisfactorily, the vaccine goes through the IND. The IND allows the vaccine sponsor to obtain permission from the FDA to distribute the vaccine across state lines to clinical investigators. At this point, the molecule being used in pharmacological activity changes legally into a new drug.
When enough human participants are found, the new potential vaccine is put through a randomized clinical trial, where people in the group are assigned to either a control or experimental group. The experimental group gets the vaccine, while the control group gets a placebo. Neither the people themselves nor the researchers know which group the participants are in. This does away with bias, and it is only at the end of the study that the researchers and participants find out who was in which group.
During phase 1 of the clinical trial, the first participants receive the vaccine being tested. The objectives of this phase are to evaluate the vaccine’s safety and its ability to produce the desired immune responses. It is often during this phase that the mode of giving the vaccine and the immunization schedule (how often to give the vaccine) are also assessed.
Once a trial has results, these results must be peer reviewed. This means that expert scientists go over the data to make sure it is correct and reproducible. There have already been multiple COVID-19 vaccine candidates where phase 1 and 2 trials stood up to peer review.
After the drug performs successfully in phase 2, it moves into the pivotal phase 3 trials, which are essential for the registration and market approval of a vaccine. These are designed to evaluate efficacy and safety. These large-scale clinical trials enroll thousands of subjects and are conducted in conditions that will be similar to the future routine use of the vaccine.
While a lot of attention has been given to pauses in COVID-19 vaccine clinical trials, these pauses are in fact signs that the system is working and science is progressing as it should, not that vaccines are dangerous. Because vaccines are given to otherwise completely healthy people, there is a high bar for trials testing vaccines to make sure they are completely safe. Therefore, if a test participant becomes ill for any reason during a trial, the trial is paused to examine the cause of the illness.
An interim analysis is done as the trial progresses in order to evaluate the effectiveness of the vaccine. In order to do an interim analysis, there need to have been enough cases of the illness among the participants to analyze the percentage that took place in the control groups as compared with the vaccinated group. The most recent COVID-19 vaccine candidate from Moderna showed a 94% success rate in its interim analysis, with only five of 95 cases of COVID-19 occurring in the vaccinated group.
articles/PMC5024796/">Bioethics in Practice: Considerations for Stopping a Clinical Trial Early, “The early-stopping rule has the potential to minimize harm and to maximize benefit for the patients enrolled in a randomized trial.”
A Biologics License Application (BLA) is required to gain permission to enter a biologic product into interstate commerce. This application requires information from the applicant, who can be any legal person or entity engaged in manufacturing, along with information about the product, preclinical study information, and labeling.
The FDA’s Vaccine and Related Biological Products Advisory Committee (VRBPAC) sets the regulatory pathway to permit wide-scale use of a vaccine and can slow down the process when it feels necessary. At the end of October, the VRBPAC held a nine-hour virtual meeting to discuss the regulatory pathway, at the end of which it told the FDA to slow down the process of trying to get a COVID-19 vaccine out so rapidly.
cially destructive diseases, such as COVID-19, some may feel that it does not move quickly enough. In that case, there is the possibility to make a vaccine available more quickly, through
Emergency Use Authorization (EUA) allows the FDA to make unapproved products, such as drug treatments or vaccines, available for use during public health emergencies. However, an EUA has never been used to administer a vaccine to civilians. This is because while drug treatments are generally given to people who are already ill, vaccines are given to people who are still healthy, so the bar for using them is much higher. In the case of COVID-19, the FDA has been considering criteria for deploying a vaccine initially under the EUA.
While the vaccines themselves are vital, they are only useful when they can get to the people who need them. That is where the vaccine supply chain comes in. Getting vaccines to millions of people requires effective vaccine handling, storage, and stock management, as well as rigorous temperature control and well-kept information systems.
In the case of COVID-19, the severity of the virus makes it especially important to get the vaccine out to as many people as quickly as possible. In the United States, states are already developing vaccine distribution plans, which deal with questions of vaccine storage, data tracking, and looking at who will pay for the vaccine to be deployed and how to make sure it is done in an equitable way.
nbcnews.com/think/opinion/getting-covid-19-vaccine-market-will-be-hard-getting-americans-ncna1237753">science skepticism and misinformation are causing people to be wary of forthcoming COVID-19 vaccines.
Also known as a phase 4 trial, post-marketing surveillance is done once the drug is already marketed and available to the general public. This phase checks the vaccine’s performance in real life scenarios, studies the long-term risks and benefits, and potentially uncovers rare side effects.
In the event of an adverse reaction to a vaccine, the CDC and FDA co-manage the Vaccine Adverse Event Reporting System (VAERS). This national early warning system accepts and analyzes reports of possible negative side effects after a person has received a vaccine. Anyone can report to VAERS.
Herd immunity happens when a virus stops spreading because it continuously encounters people who are protected against infection. This is the ultimate aim of wide-scale vaccination programs. In this case, the vaccine protects those who are vaccinated, and herd immunity protects those who are unable to be vaccinated, for example those with compromised immune systems. The unvaccinated people become less vulnerable, because the odds that they will come into contact with an infected person decrease dramatically.
Each of these questions involves some uncertainty, but it is still possible to make accurate predictions as long as that uncertainty is understood. One tool statisticians use to quantify uncertainty is called the margin of error.
I am a statistician, and part of my job is to make inferences and predictions. With unlimited time and money, I could simply test or survey the entire group of people I am interested in to evaluate the question in mind and find the exact answer. For example, to find out the COVID-19 infection rate in the U.S., I could simply test the entire U.S. population. However, in the real world, you can never access 100% of a population.
Instead, statisticians sample a small portion of the population and build a model to make a prediction. Using statistical theory, that result from the sample is extrapolated to represent the whole population.
Ideally, a good sample should be representative of the total population, including gender, racial diversity, socioeconomic diversity, lifestyle patterns and other demographic measures. The larger the sample, the more similar it would be to the true population, and with a larger sample, the more confident statisticians become in their predictions. But there will always be some uncertainty.
The larger the sample size, the more accurate the prediction and the smaller the margin of error. Fadethree via Wikimedia Commons
Quantifying uncertainty
Take drug development, for example. It is always true to predict that a new medication will be somewhere between 0% and 100% effective for everyone on Earth. But that isn’t a very useful prediction. It is a statistician’s job to narrow that range to something more useful. Statisticians usually call this range a confidence interval, and it is the range of predictions within which statisticians are very confident the true number will be found.
If a medication was tested on 10 individuals and seven of them found it effective, the estimated drug efficacy is 70%. But since the goal is to predict the efficacy in the whole population, statisticians need to account for the uncertainty of testing only 10 people.
Confidence intervals are calculated using a mathematical formula that encompasses the sample size, the range of responses and the laws of probability. In this example, the confidence interval would be between 42% and 98% – a range of 56 percentage points. After testing only 10 people, you could say with high confidence that the drug is effective for between 42% and 98% of people in the whole population.
If you divide the confidence interval in half, you get the margin of error – in this case, 28%. The larger the margin of error, the less accurate the prediction. The smaller the margin of error, the more accurate the prediction. A margin of error that is almost 30% is still quite a wide range.
However, imagine that the researchers tested this new drug on 1,000 people instead of 10 and it was effective in 700 of them. The estimated drug efficacy is still going to be around 70%, yet this prediction is much more accurate. The confidence interval for the larger sample will be between 67% and 73% with a margin of error of 3%. You could say this drug is expected to be 70% effective, plus or minus 3%, for the entire population.
Statisticians would love to be able to predict with 100% accuracy the success or failure of a new medication or the exact outcomes of an election. However, this is not possible. There is always some uncertainty, and the margin of error is what quantifies that uncertainty; it must be considered when looking at results. In particular, the margin of error defines the range of predictions within which statisticians are very confident the true number will be found. An acceptable margin of error is a matter of judgment based on the degree of accuracy required in the conclusions to be drawn.
RELATED: How vaccines get made and approved in the US