What Data? Alice In Clinical Trial Wonderland
14 CommentsBy Ed Silverman // January 4th, 2012 // 11:14 am
A remarkable series of studies published this week in BMJ discloses various problems with clinical trials - or more specifically, publication of clinical trial data. As it turns out, there is a lot that goes missing. As the BMJ editorial laments: the articles “confirm the fact that a large proportion of evidence from human trials is unreported, and much of what is reported is done so inadequately.”
Here are some of the findings: One paper examined unpublished evidence of existing meta-analyses of nine drugs approved by the FDA in 2001 and 2002, but found identical estimates of drug efficacy in only three of 41 cases, or 7 percent. Estimates of efficacy were split evenly split in the remaining cases when unpublished FDA trial data was examined (read more here).
Another example: The Food and Drug Administration Amendments Act (FDAAA) requires mandatory reporting of summary clinical trial results within one year of completion on ClinicalTrials.gov. But one BMJ paper found that only 22 percent adhered to the mandate. Generally, later phase trials and industry-funded trials were more likely to report results (look here). Another recent study found even lower compliance (see this).
Still another paper looked at trials that were funded by the National Institutes of Health between 2005 and 2008, and found that only 46 percent were published in a peer reviewed biomedical journal indexed by Medline within 30 months of trial completion. And three years later, one third still remained unpublished (please look here).
Meanwhile, another paper found that at least 3,000 records of randomized controlled trials entered into Medline between 2006 and 2011 were indexed, which means that many trial will be missed when entering the commonly used research term - randomized clinical trial - into the database. As the authors noted, important evidence in searches - design and methods, baseline characteristics, long term follow-up, and secondary data analyses - can be missed (you can read the full paper here).
Finally, yet another paper examined the potential for publication bias, data availability bias, and reviewer selection bias in recently published meta-analyses that use individual participant data. Only nine of 31 meta-analyses included individual participant data from unpublished studies in their primary meta-analysis, and the potential for publication bias was discussed or investigated in just 10. Meanwhile, 16 did not obtain all individual participant data requested, and five did not mention this as a potential limitation. Just six examined how trials without individual participant data might affect the conclusions, and reviewer selection bias was a potential issue in nine (here is the paper).
“There is an ‘Alice in Wonderland’ feel to these investigators’ efforts - acting on the public’s behalf, searching over hill and dale and among the paperwork of regulatory bodies and drug companies to put together pieces of data that should have been freely available in the first place,” the BMJ editorial chides. “…What is clear from the linked studies is that past failures to ensure proper regulation and registration of clinical trials, and a current culture of haphazard publication and incomplete data disclosure, make the proper analysis of the harms and benefits of common interventions almost impossible for systematic reviewers.”
At the end of the day, the public health implications are troubling and pervasive, an issue that is compounded when one considers that mandatory requirements - as set forth by law - are being ignored. This should serve as a wake-up call to both the NIH and the FDA to restrict grants to researchers who receive government funds, but violate the rules. This may add impetus to industry to ensure proper and timely publication, as well.
As Harlan Krumolz, a Yale University cardiologist who co-authored one of the BMJ papers, writes in his CardioExchange blog, “the era of missing data must end.”
hot tuna
Excellent article Ed, but maybe we need some feedback from Alice herself.
http://www.youtube.com/watch?v=IAOmJE46JRc
Alice
Sorry for the delay. I buried myself.
Jim
Ah, the unsubmitted data…resulting in so many deaths…such as Eli Lily’s Zyprexa and that pesky little weight gain problem.
SteveM
Re: Jim
Eli Lilly’s other über-dirty drug Cymbalata (Duloxetine) has scores of clinical trial results that were never posted.
A search on clinicaltrials.gov using; Search Terms = “Duloxetine” AND Recruitment = “Closed Studies” AND Lead Sponsor = “Eli Lilly” generates 83 hits.
Only 25 of those closed studies have results reported out, 58 do not. Meanwhile, thousands of Cymbalta side effect and discontinuation car-wreck anecdotes have piled up on the internet. With those iatrogenically battered consumers uninformed about the dangers of Cymbalta.
What nasty results from those unreported Cymbalta clinical trials did Lilly conveniently bury?
I queried FDA via email about the obligation of Lead Sponsors of registered clinical trials to report out results of closed studies. FDA responded that there is none.
Something’s wrong with the regulatory picture for sure. Yet FDA appears to be tragically indifferent.
Duane Sherry
Vera Sharav with the Alliance for Human Research Protection put out a nice post a while back…
She suggests that psychotropic drugs that were illegally marketed should no longer be paid for in Medicaid, Medicare and Veterans Affairs programs -
http://www.ahrp.org/cms/content/view/695/1/
IMO, that would be a good place to start.
Happy New Year (a few days belated),
Duane
Mindano Iha
The whole issue of clinical trials is a disgrace.
The protocols are written by the pharmaceutical industry who manipulate so that results are favourable for themselves.
Investigators are not required to register planned group and sub-group analyses.
If a protocol is discontinued the results don’t have to be passed on to the regulatory authorities.
But why are trials stopped? Because “there are difficulties”. One “difficulty” may be because some patients die!
Mindano Iha
Dr. J.Virapen:
“There should be an independent state organization which can check the protocols in advance and reject them.
If the registry required authors to be more explicit about the research plan, it would be harder to fudge data”.
original industry insider
Mindano, you are wrong on one point. When a clinical trial is prematurely discontinued a safety summary report must be written up and submitted to FDA. Furthermore these data must be included in the next IND Annual Report. Contrary to what you may have been led to believe, we don’t “bury” our deaths. We just don’t put them up in 20 point headlines in the New York Times.
original industry insider
Steve, why don’t you submit a FOIA request to FDA for all of the safety data on all of the “dirty” drugs that we in Big Pharma conspire to poison the population with. Yes, the FDA has ALL of the data, as we are required to submit these reports on a regular basis, per my response to Mindano. You most certainly deserve to receive the data as a taxpayer. With any luck, you may get a response during your lifetime if the FDA lawyerss don’t hold it up.
Suzanne
My thanks to the Agency of Healthcare Research and Quality for wrapping up all of the quality JIA studies into a neat little package for moms like me. Check out Table A for the “strength of evidence”:
http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=705#3098
Yesterday they added a pdf with comments. Also an interesting read, from my perspective at least.
http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=916
(bottom of page 10, now we know!)
I know they have to work with what they’ve got, but quit pretending there is evidence to support it.
Nick
Just to add to oii comment, in the EU deaths (and serious, unexpected adverse events) in any trial are reported to EU agencies within strictly defined time period, and all trials (including ones terminated early) must have summary report submittted to agencies.
SteveM
Re: oii - “we in Big Pharma conspire to poison the population
Cut the snark. You’re not clever enough to make it work.
Objectively, there are many Cymbalta clinical results that have not been reported out by Eli Lilly. Why?
And objectively, thousands have reported iatrogenic injury from Cymbalta and Lilly discounts the self-reports.
That’s 1+1. Are you smart enough to add it up?
original industry insider
“Iatrogenic injury”. Steve, now it’s a conspiracy between doctors who deliberately injure their patients with Cymbalta and we in Big Pharma who cover up their nefarious actions.
Sounds a bit paranoid to me. Shout out to Nick, though, to prove that there is no international Big Pharma Conspiracy, although I will admit that Pharmalot, is a fertile proving ground for paranoid conspiracy theories about my industry. I trust you feel right at hime.
Snarkily yours,
OII
WEL
MAXIDEX DEXAMETHASONE WARNING
I had eye surgery and i the post-op pack was MAXIDEX(dexamethasone) drops by ALCON LABS
Two days later I was BLIND
Use Google and enter MAXIDEX LOSS OF VISION to verify or call 800-757-9195