Benzene in your products, Part 2: The story of Valisure

Here’s Part 2 of our deep dive into benzene contamination in your products. In Part 1, we talked about whether the experiments were “realistic” (no), and whether the cancer risk was described accurately (also no). This time, we’ll be discussing the wider issues around the private lab behind these findings, Valisure.

This is adapted from the video version (which works much better than the text version, I think).

Methodology issues

We’ve covered Valisure’s questionable framing of their experiments vs reality, the risks of benzene, and the significance of their results… but what about their actual data? Can we trust it?

Well, in the past, Valisure have also submitted petitions about carcinogens in other medications – DMF in valsartan, NDMA in ranitidine (Zantac) and metformin – with a very similar vibe to their benzene ones (press releases, lots of media attention).

The FDA took their concerns seriously, they investigated… and now, there are multiple peer-reviewed papers specifically debunking Valisure’s testing methods.

And yes, you could theorise that maybe the FDA was just be embarrassed that this tiny lab showed them up, and they’re just trying to save face (I seriously considered this possibility at first). But:

1. If you look at the specific things being criticised, they’re indisputably pretty bad.

2. These FDA papers have been referenced positively in other papers, like this one, which also has authors from 11 other regulators and official testing labs from other countries – Valisure get a few shoutouts. And one author is from the Singapore HSA, who announced this contamination issue with metformin in the first place.

3. The US Pharmacopoeia, an independent scientific body who set a lot of popular ingredient standards and tests (“USP grade” refers to them), have also released a whitepaper criticising Valisure’s methods.

Being specifically called out in publications connected to this many scientifically respectable organisations is really impressive. Let’s look at what happened…

Valisure’s ranitidine (Zantac) petition

Valisure talk a lot about NDMA and ranitidine – they mention ranitidine 7 times in the benzoyl peroxide petition.

That’s because in September 2019, Valisure got their big break. They announced that they found high levels of NDMA, a probable carcinogen, in ranitidine (also known as Zantac – it’s a heartburn medication). They also said Zantac could break down inside the body to make more NDMA, and the FDA should get independent labs like Valisure to test all medication before it gets sold.

At the time, NDMA in other medications was a big deal, so this got tons of attention. Lots of countries pulled Zantac off the shelves, including the US. Valisure are pretty proud of it – and I think they do deserve some credit for pointing out the problem. The FDA took it seriously, releasing a statement saying they were investigating the problem, on the same day as Valisure’s post.

But here’s the thing. Even though their petition triggered the investigation, Valisure didn’t actually find any NDMA in normal tablets. They only found it when they used unrealistic conditions, like when they heated the samples too much (to 130 °C, or 266 °F). Heat makes Zantac break down, so you get more NDMA. So the FDA couldn’t actually use any of Valisure’s data, because their method was inappropriate and gave “artificially high” results – often thousands of times too high.

And this is a repeated theme.

Valisure vs validation

You’ll see the word “validation” a lot in these publications about Valisure. Validation means checking that your method is actually measuring what you’re trying to measure, and that your numbers are telling you something useful. Valisure just don’t seem to be great at validating their methods, and don’t seem to think they need to improve.

Measuring chemicals is pretty complicated. On TV, you chuck a sample into a machine and 3 seconds later, it tells you exactly what chemicals are in it and how much. But in reality, it takes ages, you need specific methods for every little thing, because a lot can go wrong.

Let’s say you have a perfect glass of cordial. To work out the recipe, you might mix different amounts of cordial and water, then compare how orange your perfect glass is to the glasses you’ve just made.

That’s basically how these chemical analyses work, except (note: this list is not exhaustive):

• You have a really complicated machine with lots of settings “seeing” the colour
• There’s a million other things in your sample that can mess up the machine and your results, so you have to process the sample first, which might interfere with the cordial
• The machine might think the other things in your sample are cordial
• There’s usually just a few drops of cordial in there

Lots of things can go wrong. Plus, if you’re feeling cynical, lots of things can be… nudged a bit to get higher or lower numbers. This is a big deal here, because the number tells us the dose – if something is safe or not.

With NDMA, it turns out that Valisure were using a method for a different medication (valsartan), where temperature didn’t change the NDMA level. When the FDA tried Valisure’s procedure, the samples looked burnt:

Valisure said the FDA was “so downplaying” the problem, but it’s pretty widely accepted now that high temperatures break down ranitidine to make more NDMA.

The FDA explains the importance of validation in their reply to Valisure, and how they developed and validated a new method – it has lots of helpful references, so maybe they can do it right next time.

Here’s what happened the next time…

Valisure’s metformin petition

In late 2019, Singaporean regulators found that NDMA was too high in some batches of metformin, a diabetes drug. In response, drug regulators around the world started testing metformin.

The FDA posted the levels they found in some batches of metformin, and a validated testing method. Then Valisure posted a petition claiming they tested the same batches with an “improved” version of the FDA’s method, but got different results.

Guess who found way more carcinogen?

Valisure said the FDA’s numbers were too low, they should use Valisure’s method, and… the FDA should hire independent labs like Valisure to test all medication before selling it.

Now, because this international investigation was going on, the FDA had already developed multiple validated methods, and knew what concentrations other regulators were seeing. It was obvious that Valisure’s numbers were off.

But Valisure were already talking to the media about how the FDA was wrong, and asking people to send them metformin for testing on Facebook and Twitter. They uploaded a PDF to a preprint server with a whole bunch of results, so people could check their medications (Note: Valisure have since replaced this preprint twice.) 

They also kind of mansplain to the FDA why their method is wrong.

Anyway, it turns out… Valisure’s method was wrong.

The FDA scientists tried to replicate it, and worked out that Valisure had the wrong settings on their machine. It was basically measuring another substance (DMF) together with NDMA (it was essentially counting tomato sauce as cordial). So their results were an average of 5 times too high, up to 17 times too high.

The FDA published a paper with a catchy title, and wrote to Valisure pointing out, again: you need to validate methods properly.

Valisure’s lab practices

The FDA also inspected Valisure’s lab in 2021. Their letter outlines a list of poor practices from when they were operating as a pharmacy, including not telling people about illegitimate product quickly (another disclosure issue), as well as a bunch of “scientific deficiencies” in their testing.

Valisure still seemed to be doing the same things. For example, they were using the same testing protocol for different products, which was the big issue with Zantac – and they seem to be doing this for benzene too. Different products have different things in them that could mess up your results, so the protocols need to be validated for each product individually.

And there were other issues too. For example:

• They didn’t check the calibration regularly
• They didn’t check if their preparation was breaking down the sample
• They had an incorrect baseline construction – basically the area under the peak correlates with the concentration, but you have to draw where to measure from (the red line). If you draw it wrong, you get a different number.

And this is pretty basic stuff – I’ve used HPLC and LCMS a lot, this is the kind of thing I was taught on the first day I touched the machine.

According to the letter, Valisure kind of brushed off these criticisms by saying they’re ISO accredited, so they’re not subject to GMP requirements for validation, but I think it’s really missing the point.

Validation is important for any analysis – it’s literally just making sure you’re getting good data. The reason it’s in GMP guidelines is because it’s good scientific practice. If your numbers are wrong, they’re wrong, whether you’re in a GMP lab or not.

And the FDA pointed out that these things would be required for ISO lab standards too:

Maybe despite this worrying track record, they’ve improved. But it seems like a bad idea to just assume their numbers are correct, without having other scientists and testing labs critically review and replicate their results.

What’s the harm?

So why do regulators care so much about bad data? You can probably guess, but it’s spelt out a bunch of times in these documents.

Obviously if the numbers are too low, then people are being exposed to carcinogens – the limit the FDA uses is an estimated extra 1 in 100,000 lifetime risk of cancer (0.001%). That’s super low per person, but if you’re regulating medications for millions of people, then makes a difference.

But it’s also bad if the numbers are too high. “Patients may decide to forego or change treatment” for no good reason; it can “[create] confusion and [cause] a public health risk.” Metformin is one of the most widely used diabetes treatments – if you stop treating it, well, diabetes complications are not pretty.

And the FDA aren’t even looking at the sensationalising of the results, which is what most of my posts have been about – they’re just talking about accuracy.

The USP whitepaper tackles both, and it’s the same problems:

“they can impact the behavior of health plans and patients, leading to drug shortages and can even reduce patient adherence to their treatment regimens, which can cause harm and may even be life threatening.”

And this all applies to their benzene announcements too.

I don’t think anyone is against having less benzene in products. But announcing very sensationalised, unvalidated results with tons of media attention is a really dangerous way to go about it. It blows the risks out of proportion, which means people will make bad risk assessments.

Healthcare workers avoiding hand sanitizer because of a very small increase in cancer risk, would be massively increasing the risk of infecting patients. Hand hygiene is a huge part of infection control, and sanitizer is a really useful addition to hand washing, which causes a lot more hand dermatitis.

Sunscreen is an important part of skin cancer prevention – there are other forms of sun protection, but sunscreen protects in ways and on body parts that other methods can’t do as well.

Related posts: Do Hats and Umbrellas Protect Well From the Sun?, How to Choose UV Protective Clothing

Benzoyl peroxide is a really popular acne treatment, and it’s affordable. It reduces antibiotic resistance, which is a huge problem. Acne is complicated – not everything works for everyone, people try lots of treatments before finding one that works.

Acne isn’t as life threatening as diabetes, skin cancer or hospital acquired infections, but it can make a huge difference to your life. Scaring people about life changing medications when it’s not warranted can be incredibly damaging.

Dry shampoo and antiperspirant aren’t as big a deal, but still, the fear and panic I’ve seen – people are crying in TikToks, I’ve had messages from people who haven’t been able to sleep – that’s probably way worse for your health than the tiny increase in benzene.

I’m not saying it’s a bad idea for independent labs to test things from time to time, but given their track record… Valisure do not seem like the people we should trust for this.

And yes, there have been recalls based on benzene, and NDMA. These recalls aren’t a sign that our products are killing us, which seems to be the main narrative. Products being recalled before anyone gets hurt, because of issues that aren’t expected to have serious consequences, is a good sign that our products are generally very safe. I’d be much more suspicious if there was a mysterious lack of recalls.

And notice that when health authorities announce these issues, the risk is put into context, because scaring people unnecessarily causes harm (although unsurprisingly, these got less headlines):

“Although exposure at the levels detected would not be expected to cause serious adverse health effects, and risks to users are considered low, the sunscreens are being recalled to ensure consumer safety.” – Australian TGA

“Based on exposure modeling and the cancer risk assessments published by the Environmental Protection Agency (EPA) (IRIS database), daily exposure to benzene in the recalled products at the levels detected in testing would not be expected to cause adverse health consequences.” – Government of Canada

For NDMA in metformin, the Singapore HSA stated that “the added cancer risk … is estimated to be less than 0.00002%”.

Follow the money?

So why are Valisure doing what seems to be the opposite of public health best practice?

Maybe they mean well – they honestly think these are serious public health issues that need more attention. Maybe they’re just doing all this because they’re just bad at science, but they’re really trying their hardest. Maybe they genuinely think the FDA will eventually realise they’re the right people to check product quality. Maybe they just really can’t take a hint.

And a lot of people think this:

But there’s a lot of evidence that – in my opinion – shows this isn’t the case.

I’m not a huge fan of just looking for profit motives when it comes to misinformation. People often make mistakes and are overconfident. There’s a principle called Hanlon’s razor – people usually aren’t evil.

And I’m not going to pretend I know what’s going on in the big brains at Valisure, but here are some observations…

Benzoyl peroxide announcement timing

The benzoyl peroxide announcement was on March 6 – two days before the annual AAD meeting, the largest dermatology conference. Valisure’s spokesderm friend Christopher Bunick wrote this on LinkedIn:

And conveniently, he had a speaking spot and could present Valisure’s version of “benzene science” to thousands of dermatologists, who the media and the public would listen to, and who wouldn’t have the background knowledge to question him.

“There’s a lot of confusion around how much benzene we’re breathing in the air,” according to him. I wonder why there’s so much confusion?

Three of the top people who were at Valisure have a patent application on stabilising benzoyl peroxide. That’s obviously a conflict of interest. By hyping up the dangers in front of as many dermatologists as possible, they could potentially get consumers and media to pressure companies into using their technology and artificially drive up demand.

This is a monetisation strategy with a long, illustrious history. The most famous example was probably Andrew Wakefield. In 1998, he published a study linking the MMR vaccine to autism. Most of the data was fabricated, but unsurprisingly, there was a ton of media attention. In the many interviews that followed, he kept emphasising that as a precaution, parents could choose single vaccines instead. Turns out, he had a patent on a single measles vaccine. (Brian Deer’s website has a lot of excellent documentation on this.)

Creating demand for their own product isn’t the only possible explanation. It could also be that if a company discovered a really dangerous problem, they were then motivated to research a solution.

But the patent was filed in March 2023, almost a year before the petition. It includes data on benzoyl peroxide forming benzene at 50 °C, as well as ways to adjust these formulations to reduce benzene. Experiments take time to run, patents take time to write, plus this patent includes info from 2022 (it doesn’t say which info, but you usually discover a problem before solving it).

To me, it seems like either:

• Valisure genuinely believes this is an extremely alarming health issue, but didn’t warn the public for over a year, probably two, until a more commercially convenient time; or
• They don’t actually think it’s that serious, and are just hyping up how scary it is, for… reasons

And this isn’t the first time they’ve withheld “alarming” findings…

A pattern of non-disclosures

According to the Wall Street Journal, Valisure’s president’s brother-in-law owns a law firm. They filed class action suits against Zantac manufacturers on the same day as Valisure’s announcement in 2019, which sounds pretty coordinated.

This is linked to one of their earliest unrealistic “realistic conditions”, which seems to be one of Valisure’s core personality traits at this point.

Valisure coauthored this study that supposedly showed that Zantac could produce NDMA in the stomach, and this was used in a whole bunch of law suits.

But the “stomach conditions” Valisure used had 70 to 88000 times as much nitrite as you’d find in an actual person’s stomach. Nitrite speeds up NDMA formation.

When FDA scientists replicated this study, they found that nothing happened until they reached 50 times the normal amount of nitrite in a person’s stomach. 

This quote from Valisure’s cofounder really sums up how they come across for me:

“Independent studies, even if flawed (you don’t say), that support our findings are useful.”

With dry shampoo – again, the timing is sus.

Class action lawsuits based on Valisure’s data were filed in September 2022, but Valisure only posted their results publicly in November. So it looks like Valisure probably gave these alarming results to lawyers first, and they looked for people who could bring this lawsuit months before bothering to warn anyone else.

The Wall Street Journal article also states that Unilever are getting sued over benzene in their dry shampoos, and in their court filing they say:

“Valisure sent it a proposed “agreement” to pay Valisure. Unilever would pay Valisure $1.25 million up front, plus $250,000 a month over the term of the agreement. In return, Valisure would keep its Unilever testing information confidential.”

Not telling people that you found a “known human carcinogen” if you get hush money is not exactly something I’d describe as heroic. But maybe that’s just me.

References

Yang J, Marzan TA, Ye W, Sommers CD, Rodriguez JD, Keire DA. A Cautionary Tale: Quantitative LC-HRMS Analytical Procedures for the Analysis of N-Nitrosodimethylamine in Metformin. AAPS J. 2020;22(4):89. doi:10.1208/s12248-020-00473-w

Florian J, Matta MK, DePalma R, et al. Effect of Oral Ranitidine on Urinary Excretion of N-Nitrosodimethylamine (NDMA): A Randomized Clinical Trial. JAMA. 2021;326(3):240-249. doi:10.1001/jama.2021.9199

Gao Z, Karfunkle M, Ye W, et al. In Vitro Analysis of N-Nitrosodimethylamine (NDMA) Formation From Ranitidine Under Simulated Gastrointestinal Conditions. JAMA Netw Open. 2021;4(6):e2118253. doi:10.1001/jamanetworkopen.2021.18253

Keire DA, Bream R, Wollein U, et al. International Regulatory Collaboration on the Analysis of Nitrosamines in Metformin-Containing Medicines. AAPS J. 2022;24(3):56. doi:10.1208/s12248-022-00702-4

Health Science Authority. HSA Recalls Three out of 46 Metformin Medicines. Published December 4, 2019. Accessed March 13, 2024. 

United States Pharmacopoeia. Unvalidated Methods for Medicine Quality Testing Lead to Misleading Results. August 2023.  

Valisure. Valisure Citizen Petition on Benzene in Benzoyl Peroxide Drug Products. March 5, 2024.

Valisure. Valisure detects NDMA in ranitidine. Published September 13, 2019. Accessed March 13, 2024.

US Food and Drug Administration (FDA). Statement alerting patients and health care professionals of NDMA found in samples of ranitidine.  Published September 13, 2019. Accessed March 13, 2024.

Woodcock, J. Final Response Letter from FDA CDER to Valisure, LLC (Re: Docket No. FDA-2019-P-4281). April 1, 2020.

‌Rodriguez, JD. Control of Nitrosamine Impurities in Human Drugs. Lecture presented at: Pharmaceutical Quality Symposium 2021, October 27, 2021.

Silverman, E. Pharmacy head who researched Zantac link to carcinogen says FDA has “downplayed the problem.” STAT News. Published September 17, 2019. Accessed April 8, 2024. 

US Food and Drug Administration (FDA). FDA Updates and Press Announcements on NDMA in Metformin. Published February 3, 2020. Accessed April 8, 2024.

US Food and Drug Administration (FDA). Liquid Chromatography-High Resolution Mass Spectrometry (LC-HRMS) Method for the
Determination of NDMA in Metformin Drug Substance and Drug Product. February 4, 2020.

Valisure. Valisure Citizen Petition on Metformin. March 2, 2020.

Cavazzoni, P. Partial Response Letter from FDA CDER to Valisure, LLC (Re: Docket No. FDA-2020-P-0978, metformin). May 28, 2020.

Wu Q, Evgenia Kvitko, Jessop A, et al. Analysis of Crowdsourced Metformin Tablets from Individuals Reveals Widespread Contamination with N-Nitrosodimethylamine (NDMA) in the United States (version 1). medRxiv. Published online May 22, 2020.

Light D, Kucera K, Wu Q. Comment from Valisure, LLC (on partial response to metformin petition). May 31, 2020.

US Food and Drug Administration (FDA). Spotlight on CDER Science: Rigorous Detection of Nitrosamine Contaminants in Metformin Products: Balancing Product Safety and Product Accessibility. September 3, 2020.

Throckmorton DC, Cavazzoni P. Response Letter from FDA CDER to Valisure, LLC (Re: Docket No. FDA-2020-P-0978, metformin). June 12, 2020.

Furman JP. Valisure LLC, New Haven, CT Letter (lab inspection). December 5, 2022.

Valisure. Valisure discovers benzoyl peroxide acne treatment products are unstable and form benzene. March 6, 2024.

Therapeutic Goods Administration. Multiple Sunscreens Recall – Low levels of benzene detected. May 4, 2022.

Government of Canada. Certain lots of Bed Head TIGI, Dove and Tresemmé Dry Shampoo products recalled due to detection of benzene. October 18, 2022.

Bunick CG. Discussion of new findings in benzene in BPO-containing products. Presented at: 2024 American Academy of Dermatology Annual Meeting; March 8-12, 2024; San Diego, CA.

Light D, Kucera K, Nailor J. Shelf-stable formulations of benzoyl peroxide and methods of producing same. World Patent. Published online March 13, 2023.

Deer B. Brian Deer’s 2004 film on Andrew Wakefield. Published 2019. Accessed April 8, 2024.

The Editorial Board. A Legal Shakedown Exposed. Wall Street Journal. Published February 16, 2023. Accessed March 13, 2024. 

Braunstein LZ, Kantor ED, O’Connell K, et al. Analysis of Ranitidine-Associated N-Nitrosodimethylamine Production Under Simulated Physiologic Conditions. JAMA Netw Open. 2021;4(1):e2034766. doi:10.1001/jamanetworkopen.2020.34766

Gao Z, Karfunkle M, Ye W, et al. In Vitro Analysis of N-Nitrosodimethylamine (NDMA) Formation From Ranitidine Under Simulated Gastrointestinal Conditions. JAMA Netw Open. 2021;4(6):e2118253. doi:10.1001/jamanetworkopen.2021.18253