As AI advances in NC health care systems, state leaders call for oversight
By Emily Vespa
Co-published with The Charlotte Ledger
As North Carolina’s health care systems increasingly expand their use of artificial intelligence, state leaders say they want to pioneer policy to regulate the rapidly evolving technology.
The efforts would be some of the first steps toward placing guardrails on AI in health care at the state level.
AI, which involves software that uses scads of information to accomplish tasks once thought to require human intelligence, has proliferated in health care. The advent of “generative” AI in recent years means that today, machines not only analyze large amounts of data to make predictions or recommendations, they can also output new, original content.
North Carolina health care systems have been some of the first in the country to develop and deploy generative AI technologies, such as ambient documentation tools that record patient visits and generate notes.
But these technologies come with a bevy of challenges, said Christina Silcox, research director for digital health at the Duke-Margolis Institute for Health Policy. AI algorithms can be trained on data that contains subtle biases, which ends up meaning they can underperform for patients of color, research has shown. Harnessing vast amounts of patient data can prompt privacy issues. Another danger is that health care providers could over-rely on AI tools, bypassing their years of accumulated knowledge.
Sen. Jim Burgin (R-Angier) said he plans to introduce legislation this year that would address some of the concerns about AI’s potential pitfalls. Some North Carolina health care leaders also hope to fashion an AI policy that could serve as a model for other states.
Developing policy in the complex and ever-changing AI landscape is difficult, Silcox said, but it could have enormous benefits for the future of the health care industry.
“Not only do I truly believe that AI can really improve health care and health, I also believe we need AI to improve health care and improve health,” Silcox said. “We are looking at an aging population and an overburdened workforce that’s only going to get worse.”
State AI regulation is scant
Though federal agencies have issued rules about AI in health care, Congress has stalled on creating legislation, meaning there’s limited federal oversight. Some states have stepped up to fill the void.
In Utah, for example, lawmakers in April enacted a law that requires state-licensed professionals, including most health care workers, to notify customers when they interact with generative AI. Other states have considered legislation that prohibits the use of discriminatory AI algorithms or allows patients to opt out of AI use altogether, according to the National Conference of State Legislatures.
In the past two years, North Carolina lawmakers haven’t made a similar foray into regulating AI in health care, NCSL databases of AI legislation show.
Meanwhile, North Carolina’s health care industry has amped up its use of AI. Across the state, providers have applied a host of tools to do things like help predict health events, analyze scans, handle administrative tasks and communicate with patients.
In a position statement, the North Carolina Medical Board said physicians are responsible for decisions they make at the recommendation of AI algorithms. The statement also says if physicians use AI tools to transcribe clinical notes, the onus is on them to review the notes and ensure that they are accurate.
The state medical board hasn’t yet opened any disciplinary cases that involve the use of AI tools, and it doesn’t have immediate plans to develop guidance documents or policies about the use of AI, according to Jean Fisher Brinkley, the board’s communications director.
The board is staying informed about AI but “wishes to avoid taking precipitous action that could chill innovation in the use of AI in medicine, which the Board believes holds a lot of promise,” she said in an email.
Some North Carolina health care providers have built their own governance systems for AI oversight. At Duke Health — which, alongside UNC Health, was one of the early adopters of such systems, Silcox said — AI clinical tools must go through a vetting process. A committee of system and university experts oversees the tools and their output, said Eric Poon, Duke Health’s chief health information officer.
“We ask project teams to try out the technology, deploy them with the right guardrails and see whether it works before pushing it out to the masses within the clinical population,” Poon said. The committee also scrutinizes the data that AI tools are built on for potential bias, he said.
In 2022, Duke clinicians and researchers had to revise an algorithm they were training to detect cases of sepsis, or systemic infection, in children, as reported on the TradeOffs podcast. The researchers realized that doctors took longer to order blood tests for Hispanic children later diagnosed with sepsis than white children, potentially teaching an algorithm that it takes Hispanic children longer to develop sepsis.
After investigating the issue, Duke researchers found that the AI detected sepsis in Hispanic children just as quickly as other children because there wasn’t enough data to skew the algorithm, according to TradeOffs. The researcher there, Mark Sendak, told the podcast that the incident “cemented in his mind just how easy it is for bias to slip into AI.”
This three-part NC Health News/Charlotte Ledger series explores how artificial intelligence is shaping the state’s health care.
Part one: Doctors are turning to “virtual scribes” to take notes, raising privacy concerns.
Part two: How North Carolina health care providers are harnessing AI.
Part three: How state regulators are approaching the use of AI in health care.
Still, government officials worry that leaving oversight up to the institutions is not enough.
“The question that I keep asking is, ‘AI is making all these decisions for us, but if it makes the wrong decision, where’s the liability?’” Burgin said. “Who’s responsible?”
Leaders eye a “gold-standard” state legislation
Burgin said he’s working on legislation that would address accountability when AI is used in clinical decision-making. A UNC Health executive recently indicated that the health care industry is also interested in more AI regulation.
“In an ideal world, there would be preemptive federal legislation that would apply across all states around the use of responsible AI,” UNC Health Chief Medical Informatics Officer David McSwain said at a November UNC panel on AI in health care.
That seems unlikely to happen in the near future, which means health systems will have to navigate a patchwork of state laws, he added. Scattered state laws would be complicated, he said, and could require companies to design AI platforms differently in every state.
FOLLOW OUR COVERAGE OF STATE HEALTH POLICY
McSwain said North Carolina health care leaders hope to bring together an array of stakeholders, including health systems and societies, with the goal of creating a “gold-standard state legislation” that could be replicated in other states.
A critical aspect of any future legislation would be defining terms like “AI,” which tends to be tricky because it’s used to describe a host of technologies that are used in many different ways, he said.
“I think it’s a bad idea,” McSwain said of state-level AI legislation. “But the reality of it is, that is what’s going to happen. What we want to do is establish state-level legislation that minimizes the burden on health systems, minimizes the burden on providers, minimizes the potential negative health equity impacts.”
NC Health News/Charlotte Ledger reporter Michelle Crouch contributed to this report.
This article is part of a partnership between The Charlotte Ledger and North Carolina Health News to produce original health care reporting focused on the Charlotte area.
You can support this effort with a tax-free donation.
The post As AI advances in NC health care systems, state leaders call for oversight appeared first on North Carolina Health News.
