Texas Attorney General Reaches Novel Generative AI Settlement: What Health Tech Companies Need to Know

The Texas Attorney General is once again signaling the office’s intent to enforce the state’s privacy and consumer protection laws. According to a June 2024 press release, Texas Attorney General Ken Paxton “launched a major data privacy and security initiative” to “protect Texans’ sensitive data from illegal exploitation by tech, AI, and other companies.”

In its newest action, the Texas Attorney General reached a settlement with Pieces Technologies (Pieces), an artificial intelligence health care technology company advertised as having the ability to summarize patients’ charts and notes on behalf of providers in inpatient settings, such as hospitals. The Texas Attorney General alleged that the company made false and misleading statements to hospitals about the accuracy of its generative AI hallucination rates in violation of the Texas Consumer Protection Act. Like many other state consumer protection laws, the Texas statute makes it unlawful to engage in “[f]alse, misleading, or deceptive acts or practices in the conduct of any trade or commerce.”

Specifically, the Texas Attorney General asserted that claims by Pieces that its generative AI was “highly accurate” with an error rate of less than 1 per 100,000 were inaccurate and may have misled hospital customers about the accuracy and safety of the product.

While no monetary settlement or other punitive action was involved, Pieces agreed to comply with a number of obligations for the next five years, such as adhering to prohibitions against making misrepresentations and following “clear and conspicuous” disclosure requirements.

Key takeaways for companies leveraging AI in health care include:

• Texas, like the FTC and other government agencies, is stringently reviewing AI advertising claims when enforcing consumer protection laws. Interestingly, though, this settlement did not cite the Texas Data Privacy and Security Act.
• Companies should carefully review any claims about their AI’s accuracy or hallucination rates. Under the settlement, Pieces is mandated “to clearly and conspicuously disclose (1) the meaning or definition of [any] metric, benchmark, or similar measurement; and (2) the method, procedure, or any other process used by [Pieces], or on [Pieces’] behalf, to calculate the metric, benchmark, or similar measurement used in [Pieces’] marketing or advertising of its products and services [or retain an independent third-party auditor].”
• Companies should consider including transparent disclaimers about any risks involved in using their AI tool. Under the settlement, Pieces is required to provide all current and future customers documentation that “discloses any known or reasonably knowable harmful or potentially harmful uses or misuses of its products or services[,]” including the data or model type, its intended purpose, and known (or reasonably knowable) limitations and misuses.

As we all continue to monitor regulatory and legislative developments surrounding the use of generative AI in health care, this settlement should serve as a reminder for companies and stakeholders to carefully review the accuracy of their marketing statements, while keeping consumer protection laws top of mind.

Orrick’s Cyber, Privacy & Data Innovation Group helps clients review their state and federal compliance programs, assess the impact of legislative updates on their AI and other data processing activities and update their website disclosures and internal data flows in light of regulatory guidance and litigation trends. If you have any questions, please contact the authors (Thora Johnson, Amy Joseph, Jeremy Sherer, Alyssa Wolfington, Emily Brodkin and Melania Jankowski) or another Orrick team member for additional guidance.