A surplus of positive trials: weighing biases and reconsidering equipoise
© BioMed Central Ltd 2004
Received: 5 April 2004
Accepted: 15 April 2004
Published: 27 April 2004
Skip to main content
© BioMed Central Ltd 2004
Received: 5 April 2004
Accepted: 15 April 2004
Published: 27 April 2004
In this issue, Fries and Krishnan raise provocative new ideas to explain the surfeit of positive industry sponsored trials evaluating new drugs. They suggest that these trials were designed after so much preliminary work that they were bound to be positive (design bias) and that this violates clinical equipoise, which they characterize as an antiquated concept that should be replaced by a focus on subject autonomy in decision making and expected value for all treatments in a trial. We contend that publication bias, more than design bias, could account for the remarkably high prevalence of positive presented trials. Furthermore, even if all new drugs were efficacious, given the likelihood of type 2 errors, not all trials would be positive. We also suggest that clinical equipoise is a nuanced concept dependent on the existence of controversy about the relative value of two treatments being compared. If there were no controversy, then trials would be both unnecessary and unethical. The proposed idea of positive expected value is intriguing, but in the real world such clearly determinable values do not exist. Neither is it clear how investigators and sponsors, who are invested in the success of a proposed therapy, would (or whether they should) develop such a formula.
In this issue Fries and Krishnan  raise provocative new ideas that account for the surfeit of positive industry controlled trials evaluating new drugs. Furthermore, they suggest that equipoise is a 'paternalistic' and outdated concept that should be replaced by new approaches to ethical choice in designing clinical trials and obtaining consent from potential participants.
There are two fundamental and independent concepts presented by Fries and Krishnan. First, design bias – the process of using preliminary data to design studies with a high likelihood of being positive – partly accounts for the remarkably high percentage of trials sponsored by industry that yield results favoring the sponsored drug. If design bias is indeed present, then the treasured concept of clinical equipoise, which demands that subjects entering a trial have an equal likelihood of experiencing benefits regardless of the treatment group to which they were randomized, is violated. Those authors then propose a second concept, namely that equipoise is an outdated concept and should be replaced by concepts of positive expected value (the positive sum of benefits of the two trial treatment arms), and even that subjects could enter a trial with a negative expected value as long as they are honestly informed of this likelihood.
Let us consider these concepts in order. First, Fries and Krishnan report that all 45 of the industry sponsored clinical trials presented at the American College of Rheumatology (ACR) meetings in 1 year found positive results that favor the industry product. This finding is not new, although it is more dramatic than has been seen in other investigations of this topic. In a meta-analysis of 370 trials from a large number of medical fields, Als-Nielsen and colleagues  reported that an experimental drug was found to be the treatment of choice in 16% of trials funded by nonprofit organizations, in 30% of trials not reporting funding, and in 51% of trials funded by for-profit organizations (difference P < 0.001). Indeed, the tendency of published industry sponsored trials to have positive results that favor the experimental drug has even been seen in arthritis trials. In a study that focused on trials evaluating the efficacy of nonsteroidal anti-inflammatory drugs, Rochon and colleagues  reported that industry sponsors were likely to publish results favoring their own drug.
We agree with all of the possible explanations provided by Fries and Krishnan, although we disagree with the potential magnitude of the biases discussed. For example, it was suggested that publication bias (i.e. the tendency for null studies, especially small ones, not to be published) was not a large enough problem to account for this bias toward publication of positive trials. Fries and Krishnan cite a number of sources that presumably attest to the relatively low impact of publication bias; however, our review of these references suggests that, although they are valuable publications that explore the origins of publication bias, they provide no evidence on the supposed small effect of publication bias. Indeed, much evidence is to the contrary. The initial article describing publication bias emanated from a study of ovarian cancer chemotherapy , in which it was documented that the presence of publication bias was sufficient to make ovarian cancer chemotherapy appear life saving when a comprehensive evaluation of published and unpublished trials failed to show any significant life-saving effect. Villar and colleagues  recently conducted a study in which the results of a meta-analysis evaluating the efficacy of a therapy were compared with those of a subsequent large definitive clinical trial of the same therapy. Those investigators suggested that the most prominent reason for discordance between clinical trial and meta-analysis results was publication bias in the meta-analysis. They recommended that a formal evaluation of publication bias be included in every meta-analysis so that the results will not 'mislead'.
Similarly, in arthritis trials, publication bias has been of sufficient magnitude to account for all of the reported efficacy of drugs in published studies. For example, in a meta-analysis of glucosamine and chondroitin, McAlindon and colleagues  reported that both neutraceuticals appear to have positive effects in randomized trials, but that publication bias limited definitive conclusions. Almost all of the trials included in that meta-analysis were industry funded. After publication of the meta-analysis, a large publicly funded multicenter Canadian trial of glucosamine was presented, which showed no efficacy of glucosamine, suggesting again that industry sponsorship and publication bias may account for the entire apparent effect of a therapy. In a more recent meta-analysis, Lo and colleagues  evaluated hyaluronic acid (HA) injections for the treatment of knee osteoarthritis, and reported the existence of publication bias that could have accounted for the entire treatment effect. Furthermore, that meta-analysis of osteoarthritis reported that there were three randomized trials evaluating a large molecular weight HA preparation; the two trials sponsored by the manufacturer of the preparation yielded remarkably positive results, but the one trial in which that particular preparation was a comparator against another active HA compound reported that the large molecular weight compound had absolutely no efficacy. Thus, industry sponsorship can determine the magnitude of the efficacy reported in published findings, and publication bias can account for all of the efficacy seen in published reports.
Publication bias originates primarily with the investigators, and sponsors performing trials who decide whether to submit their trial for publication. Studies suggest that it does not arise with journal editors, who are often willing to publish reports of null trials [8, 9].
Fries and Krishnan  postulate that an important reason for the positive results reported in industry sponsored trials is 'design bias'. The contention is that, given the extensive preliminary work and scientific investment in the development of a new therapy, including preliminary trials to evaluate efficacy, it stands to reason that most trials evaluating such a therapy will be positive. This argument ignores the possibility that, even when a treatment is efficacious, there may be type 2 errors (i.e. failure to find efficacy of a treatment even when it is efficacious). The likelihood of a type 2 error is directly correlated with the power of a study. In a series of studies of an efficacious agent, each with 80% power, 20% of the trials would show no significant efficacy. That all of 45 trials reported efficacy of the sponsored therapy, as indicated by Fries and Krishnan, is nearly statistically impossible, given the certainty of occasional type 2 errors.
One wonders whether all 45 trials presented as 'positive' actually had unequivocally positive results. The testing of multiple outcomes in multiple different analyses can ultimately produce a positive result when a predefined analytic approach to a single outcome measure does not. Furthermore, subset analyses can show positive results when a main effect is negative.
Another explanation for design bias relates to the choice of comparator. Both Rochon and coworkers  and Lo and colleagues  reported that a comparator drug is often selected that is 'easy to beat' and, further, that the comparator drug often performs worse with respect to efficacy than it does in other trials.
Whether the trial is designed with a weak comparator or a treatment is chosen that is nearly certain to be successful, design bias may exist; if this is the case, then clinical equipoise is absent. Fries and Krishnan  suggest that this situation is acceptable and provide alternative ways of conceptualizing the ethics of trial design that would dispense with the need for clinical equipoise.
Clinical equipoise is not necessarily only present when, as Fries and Krishnan suggest, there is a precisely equal chance of benefit with both treatments in a trial. It is present when there is a bona fide scientific or clinical controversy that needs resolution, and that is ultimately what is meant by clinical equipoise. Indeed, Freedman , who is widely credited with coining the term, defines it simply as the 'state of uncertainty about the relative merits of A and B'. This state of knowledge cannot be determined by subjects. Before approaching subjects, both researchers and institutional review boards must determine that there is such a legitimate controversy or question. If it is true that drug companies are accurate essentially 100% of the time, then there would be no controversy and therefore no justification for randomized controlled trials (RCTs). Indeed, if there is virtual certainty in the outcome of a clinical trial, then one might argue that research conducted in human subjects would be unethical. This is especially true in trials in which there is a placebo arm. Would Fries and Krishnan agree?
What about the concept of positive expected value? In the real world such clearly determinable values do not exist, and neither is it clear how investigators and sponsors, who are invested in the success of a proposed therapy, would develop such a formula. It is noteworthy, however, that if one buys this formula then it is the end of the development of 'me too' drugs. It is also interesting to consider how this would apply to drugs whose alleged benefits are that they are longer acting or more convenient to administer. What percentage would be attached to those advantages? The formula also assumes that any positive value would legitimate the research without considering that very marginal positive values may not be without risk or inconvenience to subjects.
Fries and Krishnan  propose that subject autonomy can assume precedence and that subjects should be allowed to choose to participate in a trial even if there is a negative value of treatment. This places an almost absolute value on autonomy and assumes that subject consent is the determinative fact in research ethics. However, there are values other than the autonomy of subjects that play a role. For example, an essential issue is not what subjects can consent to but what investigators can ethically ask subjects to do.
Finally, Fries and Krishnan are concerned that RCTs are the only available means by which subjects can gain access to new and promising treatments. This statement ignores the inherently coercive nature of this circumstance; the desperation of a potential subject does not provide much justification for RCTs. Also, it is not access to a 'treatment' that is at stake but rather possible access to a possible treatment – a much attenuated 'benefit'.
Ultimately, we believe that the high rate of positive industry sponsored trials presented at the ACR meetings provides an alert that either ethical problems in trial design exist or that publication and other biases allow attendees at the ACR meetings a selected glimpse of all informative trials or a biased summary or interpretation of the trial's unvarnished results.
American College of Rheumatology
randomized controlled trial