Most Failed Clinical Trials Are Never Published
32 CommentsBy Ed Silverman // September 23rd, 2008 // 8:33 am
A review of 909 clinical trials for 90 meds approved by the FDA between 1998 and 2000 found that more than half of the studies concluding a drug was ineffective were never published in medical journals. The review was published in PLoS Medicine.
The researchers combed through the FDA web site and searched published medical literature up to mid-2006 to determine if and when the results of each trial were published. What did they find? Although 76 percent of pivotal trials appeared in medical journals, usually within 3 years of FDA approval, only 43 percent of all submitted trials had been published.
Trials with statistically significant results were nearly twice as likely to have been published as those without statistically significant results, and pivotal trials were 3 times more likely to have been published as non-pivotal trials, 5 years after approval. In addition, a larger sample size increased the likelihood of publication. Having statistically significant results and larger sample sizes also increased the likelihood of publication of the pivotal trials, the researchers concluded.
“We found that there was indeed a pattern that favorable studies were more likely to be published than unfavorable trials,” Ida Sim, associate professor of internal medicine at UCSF and the lead author of the analysis, tells Bloomberg News. “This is something that is essentially structural in the way clinical trial information is disseminated to the public.”
You may recall that the recently passed FDA Amendments Act created a trial registry and requires drugmakers to submit results, which PhRMA’s Ken Johnson tells Bloomberg his trade group supports. But Sim says a brief summary of results posted on a government web site isn’t the same as full publication in a medical journal.
“Medical journals are one of the most influential ways that clinicians and the public get evidence about which drugs work or don’t,” she tells Bloomberg, partly because of the attention published studies attract from the media. And she adds the new law may give drugmakers less motivation to submit studies to journals because they can argue the web site summaries amount to full disclosure.
“If there’s less of an incentive to publish a negative study, the ratio of positive to unfavorable results might actually increase,” she warns.
The study, by the way, didn’t examine whether trials were submitted and rejected by journal editors or simply weren’t submitted at all, Bloomberg notes. Sim explains that previous research has shown the primary reason for publication bias is that companies or investigators don’t submit them.
Nathan
“Most Failed Clinical Trials Are Never Published”
Does this really surprise anyone? Are most failed experiments published on? Unless they are informative, there is no reason to publish. We’ve had this debate before. Who would publish the work? Most editors/reviewers want interesting science in their journals — not compilations of lots of failed experiments.
I’m a chemist. If I could publish papers about all the failed reactions that I’ve done in my career, I could have an encyclopedia of papers!
Dan A.
When favorable trials are published:
If I were to rate the corruptive tactics performed by big pharmaceutical companies, the intentional corruption of implementing fabricated and unreliable results of clinical trials would be at the top of the list. Pharmaceutical companies manipulate the trials they sponsor because of their power to control others involved in the process largely absent of regulation. This is a matter of requiring authenticity and, more importantly, assuring the safety of the public health.
Decades ago, clinical trials were conducted in academic settings that focused on the acquisition of knowledge and the completely objective discovery of novel medicine. Then, in 1980, the Bayh-Dole Act was created, which allowed for such places to profit off of their discoveries that were performed for pharmaceutical companies in the past. This resulted in the creation of for-profit sites, called Contract Research Organizations (CROs), which are composed of community research sites with questionable investigators void of necessary experience or quality regarding their research purpose and ability. Since they are for-profit, the trials conducted at CROs are sponsored by pharmaceutical companies that control and manipulate all aspects of the trial. This coercion is done by various methods of deception in subtle and tacit methods. As a result, research in this manner has been transformed into a method of marketing, which includes altered results of the trial to favor the sponsor’s medication. Their activities are absent of true or applied regulation, and therefore have the autonomy to create whatever they want to benefit the collusive relationship between the site and the sponsor.
Further disturbing is that once the trials are completed, the medical articles are then written by ghostwriters, who are not identified and acknowledged by the sponsor, and are not trained in clinical research overall, as they are simply freelance writers. How often ghostwriters are utilized by pharmaceutical companies remains a mystery. This activity removes accountability and authenticity of the fabricated clinical trial even further. The corruptive act is finally completed by the sponsor hiring an author to be placed on the trial that likely had no involvement with the trial, and, along with others, was paid by the sponsor. To have the trial published, the sponsor pays a journal, along with the promise of purchasing thousands of reprints of the study from the journal. Again, how often this process is performed is unknown, yet frequent enough to create hundreds of such false writers and research sites to support the industry. So benefits of medicine studied in such a malicious way can potentially harm patients and their treatment options. The purchased reprints are distributed to the sponsor’s sales force to share the content with prescribers — your doctor.
Such misconduct impedes research and the scientific method with frightening ethical and harmful concerns. Our health care treatment with medications is now undetermined in large part in such situations, as well as the objectivity that has been intentionally eliminated regarding the trust in the scientific method in this type of activity illustrated in this article. More now than ever, meds that are removed from the market are given black box warnings. Now I understand why this is occurring.
The pharmaceutical industry needs transparency and disclosure in order to correct what we have historically relied upon for conclusive proof — the scientific method. More importantly, research should not be conducted in a manner that the sponsor can interfere in the ways I described in this article. We should call for independent sites with absolutely no involvement with the drug maker. And clearly, regulation has to be enforced not selectively, but in a complete fashion. Public awareness would be a catalyst for this to occur, after initially experiencing a state of total disbelief that such operations actually are conducted by such people, of course. We can no longer be dependent on others for our optimal health. Knowledge is power, and is also possibly a lifesaver.
Ethics and Science need to shake hands.
– Richard Cabot
Matthew Holford
Nathan, you’re right: we have had this discussion, before. And there may be a level of subtlety involved in the reporting that the Average Joe is not familiar with (eg, failed because the doseage wasn’t correct, or whatever). But I hardly think it is in the mouth of the Worshipful Company to tell the public what information it may, and may not, see.
On the face of it, if there is a positive trial, one wants the drug. However, if there is a similar trial (in terms of number of participants/endpoints/etc), which is negative, then one’s perspective changes. If there are two negative trials, one moves towards skepticism.
Not publishing negative trials looks like a shitty marketing ploy, to bolster sales of a drug that people wouldn’t buy otherwise. As long as the Worshipful Company refuses to publish these trials, it will continue to look like a shitty marketing ploy.
Matt
Bio Ethics Person
If you click on the link to the study in Ed’s article you will find the conclusion:
Conclusions
Over half of all supporting trials for FDA-approved drugs remained unpublished ≥ 5 y after approval. Pivotal trials and trials with statistically significant results and larger sample sizes are more likely to be published. Selective reporting of trial results exists for commonly marketed drugs. Our data provide a baseline for evaluating publication bias as the new FDA Amendments Act comes into force mandating basic results reporting of clinical trials.
We need to know if the bias is coming from the journals, or is pharma being “too” selective in which studies they submit to publication?
It is probably a bit of both.
Anyone here have an opinion of the “FDA Amendments Act”?
Nathan
One more thing: Aren’t all clinical trial results now supposed to be deposited in a government repository of some sort? I think that is a new policy started within the last few years. It’s not quite the same as publication, but at least the information is “out there” for interested parties to dig up.
Chris
Nathan -
Depending on the study and the design, these summaries can be relatively worthless. Saying the information is ‘out there’ is incorrect in many cases. Plus there are several ways that a company can get around providing these results.
CMC guy
Nathan is exactly right that in other scientific endeavors non-successful experiments/results rarely get published (I couold have a library room not just encyclopedia). Unlike Newspaper Reporters who often focus on the negative coverage to make a story most Scientists would not last long unless they can produce positive data. Usually depending on when/what is observed a study will be halted early so often one will have incomplete data and due to constrained resources little attention gets paid to fully wrapping up and analysis. However by practice (regulation?) all “safety related information” from any clinical study is gathered and submitted to FDA. I also think Nathan is correct that such data will now be part of a government database so more widely accessible but doubt will quell all the conspiracy theorist out there.
Matthew Holford
CMC guy wrote:
“…I also think Nathan is correct that such data will now be part of a government database so more widely accessible but doubt will quell all the conspiracy theorist out there.”
Tut, tut. It is not a conspiracy theory to observe that negative trials are not published, which may impact decisions (in the case of a frontline clinician), to prescribe a drug. It is not a conspiracy theory to raise the possibility that drug reps don’t allude to the negative data, when visiting clinicians’ surgeries. It is not a conspiracy theory to observe that SAE data is suppressed. It is not a conspiracy theory to observe that whistleblowers have been bullied for trying to bring SAE data to the public domain. It is not a conspiracy theory to note that there are plenty of cases of bribery and corruption, involving quacks being paid to prescribe drugs. It is not a conspiracy theory, etc, etc.
The absence of trust in the system is not a matter of paranoia, even if there are those out there who wished it were.
Matt
Jack2
I agree with Nathan. If I could publish failed results in high impact journals I’d be a department chair in no time.
As for this study…
1. Companies need to conduct studies that don’t always address efficacy (or even safety) as part of a submission package. For example, a company may conduct a study in patients with renal impairment, and a study in patients with hepatic impairment - to determine dosing. The results of these studies are rarely published in the scientific literature, but they do go into the prescribing information to inform dosing. According to my review of the paper’s methods, these studies were included in the analysis because they fell under the umbrella of “submitted to the FDA.” These studies are also less likely to have “significant” results (which were published less often). These are also the studies likely to enroll a small number of patients (which were published less often). But that doesn’t mean there’s a deliberate and sinister attempt to hide data.
2. Among the studies that sought to determine efficacy, (the pivotal trials), a higher percentage were published (76%). These are the studies that most people care about. These are the studies that I think *should* be published.
3. I don’t doubt that in some cases the company was reluctant to publish failed pivotal studies. However, three caveats: journals would really prefer to publish positive studies - that’s what drives readership. Also, going forward, companies are legally obligated to post the results of these trials on http://www.clinicaltrials.gov. Finally, did the analysis include studies of drugs that never got approved? If so, who cares if those studies didn’t get published.
Matthew Holford
Jack2, you’re missing the point: there is no trust in the system - what part of that are you not getting?
You justifying the withholding of this information is not going to have anybody feel any better about it, especially those who perceive that they’ve experienced SAEs that the companies knew about, and suppressed. Besides which, efficacy is questionable, certainly in psychopharmacological circles, anyway - not least because they don’t know what they’re dealing with.
So, the Worshipful Company can justify its conduct in any terms that it feels are appropriate, but its product is still shite. The bottom line is that it’s making a lot of money by not doing what it’s claiming to be doing, and that’s what it’s being judged on. The only reason it hasn’t gone the same way as the square wheel is that this particular anachronism can’t be visualized for what it is by most people.
Matt
Bio Ethics Person
Jack2,
About your point three above. You say:
“3. I don’t doubt that in some cases the company was reluctant to publish failed pivotal studies. However, three caveats: journals would really prefer to publish positive studies - that’s what drives readership. Also, going forward, companies are legally obligated to post the results of these trials on http://www.clinicaltrials.gov. Finally, did the analysis include studies of drugs that never got approved? If so, who cares if those studies didn’t get published.”
It is particularly that you say: “journals would really prefer to publish positive studies - that’s what drives readership.”
Really?
This isn’t my impression. Rather, journals are more concerned with the quality of the research. This is what drives readership. Or, what should anyway.
However, journal advertising is bought and paid for by Pharma, making journals very dependent on pharma advertising dollars; which would account for some of the bias.
The study was mostly looking at drugs that were FDA approved. Yes, I would agree that studies for drugs that were not approved are not worthy of being published, I can’t imagine any pharma would waste the time and effort to submit them either.
So, if this report states that over half of the clinical trial data for approved drugs is unpublished as far out as five years after FDA approval, then we have to ask why?
Interestingly, important trial data is showing up “after the fact”in a few “famous” cases, such as Vioxx, Vytorin, etc….
If the information exists, it should be accessible to and by everyone.
Anne
And another “famous case”…..Zyprexa. Thanks to Lilly hiding lethal side effects and manipulating clinical trials, my son was killed.
Yet you tell me these men will not go to prison.
Jim
If the product is approved the studies should be required; if they are not then there is no reason to publish the results.
Jack2
BEP: Journals like JAMA have made an effort to try to judge studies only on the merits of the quality of the study. However, there’s still a bias to only publish positive results. For clinical research look at NEJM or JAMA. For basic research look at Science or Nature. Count the number of positive studies and count the number of negative studies. You will see an overwhelming majority of the studies are positive. It’s just easier to publish positive results.
Matthew Holford
Jack2 wrote:
“…It’s just easier to publish positive results.”
Well, these “academic articles,” are viewed by the industry as a marketing tool, I understand, and not for the purposes of conveying information to clinicians. In that context, the bias towards positive studies isn’t that surprising.
Matt
Nathan
The assumption here is that “failed clinical trials” are not published because they somehow present negative data. That’s not necessarily the case. Pharma frequently tries to extend the utility of their FDA-approved products into new indications. This, of course, requires clinical trials to support the new indication. If the trial for the new indications fails (which it often does), then of course the results will never be published. They will, however, be deposited into the online database that Jack2 pointed out. I don’t think most people would find this practice objectionable.
Nathan
Per my above comment, remember: It is much MUCH cheaper to add a new indication to an existing drug than it is to developed a new drug for a new indication. (assuming that there is adequate patent life remaining on the “old” drug) That’s what drives the practice of “fishing” for all these new indications. Hence you have lots of failed trials for drugs on the market.
Nathan
Matthew writes: “Well, these “academic articles,” are viewed by the industry as a marketing tool, I understand, and not for the purposes of conveying information to clinicians.”
Matthew, have you ever cracked open an issue of JAMA, NEJM, Science, or Nature? Just wondering if you are speaking from experience or just your “intuition”…
Chris
Nathan -
If the off-label use of a drug has mixed results depending on the design of the trial, and the negative results are not published and the positive results are published, how is a clinician supposed to objectively evaluate off-label use?
truthman30
What amazes me about all this is..
A pharmaceutical company can do 99 trials on a drug and they could all be negative, but one positive study could still get the drug passed..
Seems like they just keep testing it until they get the results they want..
And if that fails then they just change the study end points and pretend it was effective..
Nathan
Truthman,
Re-read the comments above. The first 99 “failed” trials may have looked at the wrong indication, the wrong dosage, the wrong endpoints, the wrong formulation, the wrong dosing schedule, or the wrong sub-population. Failing a clinical trial does not mean that the drug is a failure. Instead, it means that the specific hypothesis being tested by the drug was not correct.
Bio Ethics Person
Jack2 and Nathan,
You are both assuming that the studies that are not being published are “negative”. Which of course, throws this discussion thread off track.
However, it does clearly illustrate some “pharma thinking” here.
The point is, studies should not be viewed by whether they are positive or negative, it is the information they provide which is important. For example, if a study shows that a certain drug should not be used for a certain indication, then it is the information learned in the process of the study that is important for the clinician and the patient who wants to be fully informed.
Just A Thought
Doctors don’t get to choose their patients like a well selected jury. If studies have failed in that wrong sub-population, they should be informed. No?
Matthew Holford
Nathan wrote:
“…Matthew, have you ever cracked open an issue of JAMA, NEJM, Science, or Nature? Just wondering if you are speaking from experience or just your “intuition”…”
Oh, gee, Nathan - you got me! I don’t subscribe to any of the esteemed organs that you mention.
But the marketing “professionals” certainly understand the value of a clinical trial:
http://www.researchandmarkets.com/reports/301981/successfully_marketing_clinical_trial_results
Matt
Nathan
Just a thought writes: “Doctors don’t get to choose their patients like a well selected jury. If studies have failed in that wrong sub-population, they should be informed. No?”
Absolutely - I agree. It would be unethical and probably illegal to not provide such relevant information to the FDA in order to compose labeling info. However, publishing in a scientific journal is another story. I agree it makes sense to do what you suggest when there is relevant negative info about a subpopulation. I doubt that very many people in industry would disagree with you. I also doubt that this scenario accounts for a very large percentage of the unpublished work. Also, you have to consider another point that someone made above: Many times trials are halted when such side effects or lack of efficacy is observed. This constitutes an “incomplete” experiment. What journal would publish such material?
Matthew Holford
Sorry, it was this one that I meant to post:
http://www.clinicaltrialstoday.com/2008/08/merck-report-co.html
Matt
Matthew Holford
No, hang on, it was this one:
http://www.marcusevans.com/html/eventdetail.asp?sectorID=32&EventID=11612
Matt
Matthew Holford
Nathan wrote:
“…Many times trials are halted when such side effects or lack of efficacy is observed. This constitutes an “incomplete” experiment. What journal would publish such material?”
I’ve no idea, but it should certainly be available if a drug is licensed, based on successful (dare I say “cherry-picked”? Clearly I do), trials. In the UK, only the regulator gets to demand all trials data. How is a frontline clinician supposed to prescribe accurately, on that basis? And what hope has the patient got?
Look, Nathan, the bottom line is that there’s no trust left, and the Worshipful Company has brought that state of affairs upon itself. You may argue all you like about the propriety of not publishing this negative material, but that won’t change anybody’s perception.
Matt
truthman30
Nathan you seem to consistently miss the point here..
Whether you do that purposely or it goes over your head I have no idea…
So maybe, I should make it clear for you..
If for example a drug company does 5 clinical trials for a new drug…
And let’s say 4 of them failed and the results were negative in terms of efficacy , side effects , effectiveness etc, and lets say one study performed slightly better than placebo and this study is the only one published..
Does that mean the drug is safe? Does that mean efficacy has been proven? Does that mean that the drug should automatically hit the market? ..
Of course it doesn’t, what it actually proves is that the drug failed more times than it passed…
It’s bad science, fluke and pot luck..
Paxil is a perfect example of this kind of thing…
truthman30
If Ford made a car that failed in 9 safety tests but passed marginally in 1 would you drive it?.. Would you buy it?… Would it be safe?…
No..
It would be dangerous…
It’s the same with drugs…
Just A Thought
Nathan, what I meant was that clinical testing can omit high risk participants- if sufficient testing is done at all.
My concern is with ANDAs for the most part. Some may consider them to be less important than NDAs, but being a chemist you know that changing any formula creates new risks.
So when you say:
“It would be unethical and probably illegal to not provide such relevant information to the FDA in order to compose labeling info.”
I say that- for a reformulated branded drug to be approved by the office of generics, based on a couple of one dose studies, does not allow for the FDA to make a fully informed decision. And they seem perfectly okay with that.
What happens to consumers if a new sub-population is created through the reformulation of a drug? Or rather, what if a known sub-pop is ignored by these very limited studies?
Is it ethical then?
Our doctors do not get to select only patients without a narrow therapeutic index. And they sure don’t get to give their patients only one pill. The very least they should get is information so that they know that what was successful yesterday may not work tomorrow. With ANDAs they do not get that information and it is dangerous.
I don’t know where that falls under the term ‘ethical’ but it does make our drug supply untrustworthy.
Mr. Blue
The problem of failed trials will decrease as we move away from ccreating 10,000 chemical compounds like shooting a massive spray of buckshot, and hoping that one lands.
Genomics will play a huge part in that, as we will be able to determine
1) Which patients will even respond to the product, and
2) Engineer the product to target specifcally those individuals.
And, if we don’t move more toward a European style of conducting and approving clinical trials, this industry will comprise 95% of our GDP within the next 20 years or so.
Sound nutty? Print and save.
Mr. Blue