Bias In The Lab? Really? Mark Lindner Explains…
20 CommentsBy Ed Silverman // July 21st, 2008 // 11:56 am
As drugmakers race to discover and develop new drugs, a nagging question may linger – was the proper criteria used to move that compound from the pre-clinical to the clinical phase or was there some bias involved? Mark Lindner, a PhD in pre-clinical behavioral pharmacology and consultant who was previously a principal scientist at Bristol-Myers Squibb, says the need for better controls at the pre-clinical phase would lead to better choices and, ultimately, save big pharma big money. He published a paper about this issue a year ago (here it is) and we recently spoke with him about the problem, which he believes is industrywide…
Pharmalot: How big a problem is the bias you desribe?
Lindner: You can look at the literature to see. The FDA requires clinical trials to have special controls – double-blinded, randomized, placebo-controll trials - to prevent bias. But it’s not required in pre-clinical, so people just don’t do it. I’m not suggesting pre-clinicals are sloppy or aren’t well designed. People do use control groups and careful measurements, but the extra controls to prevent bias are not used as a standard precaution.
Pharmalot: What kind of bias are we talking about?
Lindner: There are a lot of different ways, but it’s important to emphasize that I’m not talking about fraud or willful deceit. I’m talking about things that affect the decision-making process and judgement outside your usual awareness. But the assessment of a compound’s therapeutic potential are made by drug discovery teams, which are responsible for advancing the compound into the clinic. And they’re being asked to justify a way to advance that compound. So they’re very motivated, because they’re expected to be an advocate.
As an example, take a compound for treating Alzheimer’s. The best way to test it is with test A. But if it doesn’t pass, then you go to test B. You can keep going down the list to less clinically relevant tests until you get the result you want it to show. And that’s commonplace in industry. In order to drive your program, you need a way to distinguish your compound. So you have to use something that shows an effect.
Pharmalot: Sounds like there’s a lot of waste.
Lindner: The job of pre-clinical groups is to advance compounds. But it’s diffcult for more senior executives to decide what to take into the clinic. It’s hard to distinguish when everything is presented as a strong candidate, because every team puts together a data package that’s as strong as they can make it. It becomes more of a promotional package.
Pharmalot: But each group focuses on a very few compounds out of many, right?
Lindner: Yes, they may look at thousands, but ultimately, only choose one, two or three. Look, this system evolved for certain reasons. And they were good reasons. The effort makes sense if people are focused on completing an objective and making sure people are motivated. But there are reasons to be reworked.
Pharmalot: What’s the practical effect?
Lindner: This is all about why things look efficacious in the models, but not necessarily in the clinic…There’s so much attention now on why the industry is struggling – you hear a lot about the need for innovation. But I’d argue one of the most important elements of the whole process that has been neglected is the human element. People just don’t think like computers. That’s why you get bias. You need to address the role that scientists play in the process.
Pharmalot: And so the solution would be what?
Lindner: You need to have more controls in the pre-clinical phase. Otherwise, you’ll continue to have those issues. I think that’s the change that needs to happen. It’d be difficult, but it’s inevitable that some companies will realize these issues need to be addressed, because they cost a lot of money. I’m hoping people in charge of success rates will be open to the issue.
Dan
Recently, I read a similar and related article related to bias in journalism on plos.org. Thier recommendation to reduce bias and increase objectivity in both cases, among other recommendations, is to have external monitors and reviewers without any relation to those involved in both trials and motives for reporting results from such trials.
And I agree this is necessary and should have been mandated from the beginning.
Nathan
Lindner writes: “take a compound for treating Alzheimer’s. The best way to test it is with test A. But if it doesn’t pass, then you go to test B. You can keep going down the list to less clinically relevant tests until you get the result you want it to show.”
He’s absolutely correct about we evaluate compounds in early R&D. The thing he is missing, however, is that we don’t always KNOW what the “clinically relevant tests” are. For example, take something that I’ve worked on for quite a while: Asthma/allergies. The most “clinically relevant test” out there is the sheep airway hyper-responsiveness model. Even though this model is considered the most clinically relevant model, some common allergy medications fail to work in this model (including Singulair!). Where does this leave us? Is it still a clinically relevant model? Yes. Should we kill a compound just because it fails this model? Not necessarily.
Overall, I agree with what he is saying. It is hard to know sometimes whether I am being biased in my interpretation of the preclinical models we are using.
Protistic
The idea that preclinical scientists can just keep trying different tests until one works, AND that the senior management who make the multi-million dollar decision whether to advance a compound into further development will blithely accept such a data set is just silly. A scientist who presented 5 negative models along with the one that the compound worked in as the basis for a development decision would have a (very) short careeer.
Almost as silly is the idea that preclinical assays don’t include controls.
Needless (?) to say, neither assertion is present in Linder’s article, which focuses on the potential distortion of preclinical results by (conscious or unconscious) operator/scietist bias. Perhaps unsurprisingly it is only when Lindner’s comments are distilled into sound-bites throught the journalistic filter that his ideas come across sounding silly or trivial.
But the real question here is whether Lindner’s actual thesis, that bias in preclinical assays must or even should be subject to the same level of scrutiny as the FDA subjects clinical trial designs, is a valid concern.
First, is there a valid regulatory interest in preventing companies from wasting their money on wild goose chases? Rather, the regulatory concern is primarily in assuring the safety of research subjects. That is done by mandating rgorous preclinical toxicology and safety pharmacology evaluations, and those assays are in turn regulated under quite stringent Good Laboratory Practice (GLP) regulations. A secondary regulatory concern is to assure that human subjects have some reasonable possibility of beneficial outcome. For the earliest clinical trials in most therapeutic clases the earliest studies are done in healthy volunteers - no disease; no possibility of a beneficial outcome. In the case of studies which do use patients in the earliest trials (e.g. oncology), there does need to be some expectation of benefit; my experience is that in such cases regulators reviewing IND’s and IRBs reviewing protocols are quite sensitive to the strength and rigour of the preclinical package.
Protistic
Sam
I had to laugh at the first sentence of this story - As drugmakers race to discover and develop new drugs” The new drugs Caduet - Pfizer’s
combo of Norvasc and Lipitor. Pfizer’s Viagra and the new drug
Revatio both drugs are sildenafil. The old drug Flexeril and available
cheaply as cyclobenzapine is now available as Flexmid and Amrix, but
at just different doses or delayed-action and with hugh high prices.
Then there is Simcor (niacin and Zocor) and Advicor (niacin and Mevacor)
We cannot forget Vytorin (Zetia and Zocor)
Lindner
Nathan
Preclinical scientists usually have a reasonable idea about how to rank-order the models with respect to their clinical relevance. After all, there is a reason for why compounds are tested in certain models first. A drug that fails to show efficacy in the ‘best’ model, or the ’second best’ model, and so on, may still be effective in the clinic, and I’m not arguing that those compounds should necessarily be dropped. I’m only arguing that senior executives, deciding which compounds to advance into the clinic, should be aware of how the models are rank-ordered in terms of their apparent clinical relevance, and all the data that was collected should be presented to them, not just the positive data. Decisions about what to advance into the clinic should be based on a thorough, independent, objective review of all the data, positive and negative. The standard procedure now is for leaders of each drug dicovery team to put together the strongest package they can, in order to persuade senior executives to advance their compound. Team leaders are expected to be advocates for their program and they are rewarded for advancing compounds through the pipeline and into the clinic. This system, by its very nature, leads to biased reviews and poor decision-making, and accounts in part for why so many things seem to look efficacious in the models only to fail in the clinic. There is a tremendous amount of research on human judgment and decision-making that could be brought to bear on drug discovery research that would help improve clinical success rates and increase return on investment.
Ed Silverman
Dear Protistic,
For better or worse, I attempted to engage Dr. Lindner in a serious discussion about his findings and the problems he sees with drug development.
And I take full responsibility for the tone and direction of the discussion. However, at no time, were his comments “distilled into sound bites through a journalistic filter.”
I asked certain questions out of curiousity and he responded in a way that was meant to be helpful and, hopefully, informative.
You’ve raised some good points and I think Dr. Lindner is better qualified to respond than me. And of course, you’re free to disagree, but please don’t assume that there was an attempt to skew his thoughts.
Regards
ed silverman
Lindner
I have to agree. I think Ed did a good job to read a very lengthy paper and to ask relevant questions, and I think he did a good job communicating my responses to his questions.
Thank you Ed, and keep up the good work!
http://www.lindnerpreclinical.com/
Melody
neither assertion is present in Linder’s article, which focuses on the potential distortion of preclinical results by (conscious or unconscious) operator/scietist bias.
How does misinterpretation of the data play into this mix? In early trials testing rDNA human insulin, the scientific ‘conclusion’ was that rDNA ‘human’ insulin was a more effective insulin than natural animal insulin BECAUSE it took more exogenous glucose to counteract excess human insulin. The researchers failed to address (1) that the test subjects were non-diabetic, and thus actually producing their own endogenous insulin, and (2) because rDNA human insulin does not cross the blood-brain barrier and stimulate counter-regulatory co-factors that elevate blood sugar, rDNA insulin did not produce these co-factors and resultant elevated bGs.
This may seem like a “plus” for the outsider, looking in . . . no more sweats, shakes, dizziness, confusion, etc. But, for a diabetic dependent on symptomology to alert for impending hypoglycemia, those “missing” symptoms are extremely important. Of course, we saw the mushrooming of the monitor/strip business to ‘protect’ diabetics from this dangerous new drug . . . and the researchers ‘proved’ what Lilly needed to have proven to garner FDA approval. This single ‘misinterpretation’ of the data, whether intentional or not, has had a significant impact on diabetics . . . but has also served as a driver for diabusiness, increasing costs (via increasing insurance premiums) for everyone.
Unintended consequences of misinterpreted ‘results’ can be hell!
Jack2
Melody,
Does normal insulin cross the BBB? Or does c-peptide cross the BBB and affect the brain?
Unlike the rampant cynicism and irreconcilable differences common on this site, I’m genuinely curious.
Just A Thought
I’m curious too if “normal insulin” is now impossible to get or if it is simply not paid for by most insurers, or what? It’s so hard to understand not allowing for an option when something works for so many.
Doesn’t that mean we’re going backwards? That progress is anything but?
Nathan
Lindner writes “Preclinical scientists usually have a reasonable idea about how to rank-order the models with respect to their clinical relevance”
First, thanks for your response. Like I said, overall I agree with you. There is a need to get a handle on our biases that we build into the system. But I’m not sure I totally agree with your above statement. We have a good idea how to rank-order our models in terms of which we LIKE the best — but we don’t really know which is most predictive of what we will see in man. Animal models are always evolving. They are notoriously “noisy” in terms of their reproducibility. Newer models may be MORE predictive of clinical relevance – they just haven’t been around long enough to show their success.
As a project team leader a big-pharma, my job is to push the best compound we can into the clinic and see if it works IN MAN. We can cure almost any disease in a mouse. But mice are very, very different than man. Success in animal models are only the “admission ticket” to the great lottery we all call “clinical trials”.
Melody
Jack2–
Natural (animal) insulin DOES cross the blood-brain barrier . . . and when insulin level is too high, the brain can signal counter-regulatory responses. Because rDNA HUMAN insulin does not cross the b/b barrier, when a diabetic has a significant drop in blood glucose, there is no symptomology (except the ever-present bG monitor that is a mainstay of current diabetes treatment) that allows the individual to know they are in serious, grave, or life-threatening territory.
Just A Thought–
Natural animal insulins (derived from bovine or porcine sources) are no longer available in the United States. Individuals who cannot survive on rDNA genetically-engineered insulin (and those who appreciate a better quality of life on these products) have a convoluted pathway of personal importation that (if affordable) allows for purchase from a foreign manufacturer.
Interestingly, as the brouhaha heats up regarding generic biotech entrants into U.S. markets, rDNA Human Insulin was approved under small-molecule guidelines, and remains so classified today. Nevertheless, it has the distinction of being the first genetically-engineered (biotech) drug. While branded manufacturers argue that generic biotechs face the very high bar of proving identical-ity, the branded manufacturers, themselves, have difficulty proving the same thing from batch-to-batch. Insulin is particularly troubling since all batch-testing requirements were eliminated over a decade ago. Bottom line seems to be that the “free market” is not necessarily a “fair market”. In the unique case of insulin, freedom of choice is non-existent because the insulin manufacturers CHOSE to remove it from the marketplace to force consumption of their genetically-engineered products.
You might find the following article interesting: “What’s In Your Bottle of Insulin” at http://alliesvoice.com/2008/03/11/allies-voice-whats-in-your-bottle.aspx or “Innocent Until Proven Guilty at http://sstrumello.blogspot.com/2008/07/innocent-until-proven-guilty.html#links
Lindner
Nathan
I think we do agree with each other for the most part, I’m just trying to make the finer points of my argument as clearly as I can. I agree with all your caveats about the models. My point is that the assessment of therapeutic potential, especially given all the subjective aspects of the models, should be handled by people who are charged with being objective, and who are rewarded based on the ultimate, overall clinical/FDA approval rates. Assessments of therapeutic potential should not be handled by project team leaders such as yourself, who are required to be advocates for your compounds, and who are rewarded for getting your compounds into clinic trials. No matter how conscientious you may be, a considerable amount of research shows that cognitive functions outside of your conscious awareness and also outside of your control, will affect your judgment and decision-making abilities. I am trying to change the way preclinical assessments are conducted, to capitalize on Nobel prize-winning research on human judgment and decision-making by Herbert A Simaon and Daniel Kahneman, to dramatically improve clinical approval rates and return on investment.
http://www.lindnerpreclinical.com/
Just A Thought
Sorry to hear it Melody. I do understand.
Jack2
Hi Melody
In a person without diabetes I understand the correlation between insulin and glucose. But isn’t this somewhat broken in a person with Type 1 diabetes?
For example, a person without diabetes could eat 0 snowcones or 10 snowcones. If they eat 10 snowcones they will have a huge spike in insulin, and if they eat 0 they will have essentially zero spike. If a diabetic eats 10 snowcones or 0 snowcones they will not get an endogenous insulin response. So the amount of insulin in his/her body hinges on the exogneous insulin injected.
Again, I’m not trying to antagonize, just understand. I don’t understand how the brain “sensing” or not “sensing” insulin that crosses the blood brain barrier necessarily allows a diabetic to “feel” blood sugar concentrations.
Melody
Jack2–
Let’s forget non-diabetics. A well-functioning endocrine system handles 0 snowcones or 10 snowcones. Instead let’s look at a diabetic who uses natural insulin (that crosses the b/b barrier) and the same individual forced (no choice) to sustain his/her life with synthetic insulin.
Using natural insulin, let’s suppose I (1) take too much insulin, (2) get too much exercise, (3) eat too little (or no) food. When my brain recognizes that my system has TOO MUCH INSULIN, it signals for counter-regulatory measures that stimulate my liver to begin converting stored reserves into usable fuel. This conversion is not achieved quickly, so, at the same time, my brain gives me warning signals (shakes, sweats, dizziness, confusion) that I RECOGNIZE and act on . . . I get myself some fast-acting carbohydrates to alleviate the impending danger. OJ, cola, sugar cubes, etc. will begin almost immediately (within 10 minutes) to alleviate the SIGNALS. The shakes subside, the sweating lessens, the brain fog begins to clear. My exogenous ’sugar’ supply is acted upon by the overabundance of insulin. Assuming that I have access to food, the warning signals help to prevent me from passing out and being unable to intervene on my own behalf and save my life.
Same scenario–on synthetic insulin, which does not cross the b/b barrier. I DO NOT get the shakes, sweats, brain fog, etc. If I am lucky . . . or if I use my bG monitor with regularity (every 1/2 hour to 1 hour), I can see by “the numbers” that I need to take a remedial action. If my monitor flashes a 20 or 30 or 40 at me, I can head for the refrigerator—even though I have NO SYMPTOMS of low bG. If I don’t test with great frequency . . . or if the situation occurs while sleeping, I cannot take any action to alter the situation. If I am lucky, and the excess insulin is not TOO GREAT or does not act TOO QUICKLY, the counter-regulatory measures of my autonomic system will indeed engage, and I may not die. On the other hand, if I do not know (”feel”) that my bG is too low, I may die before my body can satisfactorily counter-regulate the over-insulinization. So, despite the ugliness of the warning signals, I prefer to be able to act on my own behalf rather than relying on “luck” to rescue me. Unfortunately, when Eli Lilly removed my choice—I now have ONLY various formulations of synthetic insulin or analogs—they removed my safety net. I am now captive to a bG monitor (or at least will be when I can no longer import natural insulin from a foreign manufacturer.)
You know what shambles the AE reporting system is, and how little time doctors have to talk to their patients as well as to their peers. We believe that epidemic numbers of dead-in-bed victims, sudden death victims and victims of serious auto accidents are the result of these non-warning, quick- and high-peak insulin surges. When recognized, these events are labeled as hypoglycemia unawareness events. Sadly, many (most) are not recognized and merely attributed to non-compliance. Very young diabetic children and older Type 2 diabetics being introduced to insulin therapy are probably among the most vulnerable—they don’t know what they don’t know.
Jack2
I think I see your point, but if you have type 1 diabetes the TOO MUCH INSULIN signal you mention might not mean too much insulin. If you eat 10 snowcones, your sugar could be so high that what your brain thinks of as TOO MUCH INSULIN might actually be the right amount of insulin, since you need so much.
In a person without diabetes there’s a connection between insulin and glucose (glucose goes up - insulin goes up afterwards). In someone with type 1 diabetes this natural connection is broken. Your endogenous insulin level, the biggest single hormone involved in glucose regulation, is essentially zero - it’s determined by your exogenous insulin level.
But almost all the short-term complications come from low glucose, and almost all the long-term complications come from high glucose. The complications don’t come from insulin level (beyond it’s affect on glucose) - insulin levels are secondary.
The insulin level in someone with type 1 diabetes correlates with injections, not glucose levels. Because of this disconnect, I don’t see how the brain sensing insulin can confidently let you know there is or is not something wrong with your insulin - since the two levels don’t necessarily follow one another in a person with type 1 diabetes (although they will under ideal treatment conditions). A high insulin level could be the right level, and a low insulin level could be the right level, depending on the glucose level.
It’s possible I’m missing something here. You’ve obviously thought about this much more than I have. But mechanistically, I don’t understand why you wouldn’t need to monitor your glucose level with either type of insulin.
Jack2
This paragraph should read…
The insulin level in someone with type 1 diabetes correlates with injections, not glucose levels. Because of this disconnect, I don’t see how the brain sensing insulin can confidently let you know there is or is not something wrong with your GLUCOSE - since the two levels don’t necessarily follow one another in a person with type 1 diabetes (although they will under ideal treatment conditions). A high insulin level could be the right level, and a low insulin level could be the right level, depending on the glucose level.
…when can we get an edit button around here?
Melody
The brain relies on blood glucose to survive and keep all biologic systems operating. The brain per se does not sense high bG levels. Whether T1 or T2, insulin insufficiency or bad receptors do not trigger alarms.
However, significant drops in bG levels in the brain will—slowly or quickly—signal the brain to enter into a flight-fight response that causes the release of cortical steroids, epinephrine, adrenaline, etc, which causes the EXCESS INSULIN reaction.
You are correct: the brain can cause a ‘false’ insulin reaction when bG levels drop from a high level to a significantly lower level in a short period of time. The lower number may be ‘normal’ or even above normal, but the brain sensed the cascading drop within a short period of time.
For some diabetics, the brain sits fat and happy with normal bG longer because the blood bG lowered by excess insulin was not lowered as quickly in the brain due to the insulin’s inability to cross the blood-brain barrier. Natural insulin is lipid-soluble, crosses the b-b barrier with ease and lowers the brain cell bG level at the same time the action is occurring in the circulating blood. This triggers ‘early’ warning signals that allow a diabetic to ACT, rather than waiting for the body to re-act—perhaps too little and/or too late.
The need to monitor is now mandatory because of the nature of ‘modern’ insulins. Prior to the early 1980s, diabetics survived, often long-term and without complications. Modern insulins and technology are haled as progress. In truth, they are merely drivers of today’s diabusiness and enslave diabetics. Ask most long-time diabetics (those of 35+ years’ duration) if they have experienced improved quality of life, lessening of complications or increased expectations of longevity with all this PROGRESS and I except you will find more who answer NO than YES.
Here are a few links that provide further information:
http://www.sfn.org/index.cfm?pagename=brainBriefings_bloodBrainBarrier
and
http://alliesvoice.com/2008/06/25/allies-voice-preventing-diabetes-complications-with-nature.aspx
and
http://alliesvoice.com/2008/06/10/allies-voice-dirty-whitelies-big-pharma-told.aspx
Jack2
I still don’t know exactly what insulin does in the brain. When I think of insulin functions I think of the liver not the brain. However, I see there are endogenous transporters that specifically transport insulin across the BBB into the brain, so I have to concede it probably does something.