This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

Every year, some 22,000 Americans a year are killed as a result of serious medical errors in hospitals, many of them on operating tables. There have been cases where surgeons have left surgical sponges inside patients’ bodies or performed the wrong procedure altogether.

Teodor Grantcharov, a professor of surgery at Stanford, thinks he has found a tool to make surgery safer and minimize human error: AI-powered “black boxes” in operating theaters that work in a similar way to an airplane’s black box. These devices, built by Grantcharov’s company Surgical Safety Technologies, record everything in the operating room via panoramic cameras, microphones in the ceiling, and anesthesia monitors before using artificial intelligence to help surgeons make sense of the data. They capture the entire operating room as a whole, from the number of times the door is opened to how many non-case-related conversations occur during an operation.

These black boxes are in use in almost 40 institutions in the US, Canada, and Western Europe, from Mount Sinai to Duke to the Mayo Clinic. But are hospitals on the cusp of a new era of safety—or creating an environment of confusion and paranoia? Read the full story by Simar Bajaj here.

This resonated with me as a story with broader implications. Organizations in all sectors are thinking about how to adopt AI to make things safer or more efficient. What this example from hospitals shows is that the situation is not always clear cut, and there are many pitfalls you need to avoid.

Here are three lessons about AI adoption that I learned from this story: 

1. Privacy is important, but not always guaranteed. Grantcharov realized very quickly that the only way to get surgeons to use the black box was to make them feel protected from possible repercussions. He has designed the system to record actions but hide the identities of both patients and staff, even deleting all recordings within 30 days. His idea is that no individual should be punished for making a mistake.

The black boxes render each person in the recording anonymous; an algorithm distorts people’s voices and blurs out their faces, transforming them into shadowy, noir-like figures. So even if you know what happened, you can’t use it against an individual.

But this process is not perfect. Before 30-day-old recordings are automatically deleted, hospital administrators can still see the operating room number, the time of the operation, and the patient’s medical record number, so even if personnel are technically de-identified, they aren’t truly anonymous. The result is a sense that “Big Brother is watching,” says Christopher Mantyh, vice chair of clinical operations at Duke University Hospital, which has black boxes in seven operating rooms.

2. You can’t adopt new technologies without winning people over first. People are often justifiably suspicious of the new tools, and the system’s flaws when it comes to privacy are part of why staff have been hesitant to embrace it. Many doctors and nurses actively boycotted the new surveillance tools. In one hospital, the cameras were sabotaged by being turned around or deliberately unplugged. Some surgeons and staff refused to work in rooms where they were in place.

At the hospital where some of the cameras were initially sabotaged, it took up to six months for surgeons to get used to them. But things went much more smoothly once staff understood the guardrails around the technology. They started trusting it more after one-on-one conversations in which bosses explained how the data was automatically de-identified and deleted.

3. More data doesn’t always lead to solutions. You shouldn’t adopt new technologies for the sake of adopting new technologies, if they are not actually useful. But to determine whether AI technologies work for you, you need to ask some hard questions. Some hospitals have reported small improvements based on black-box data. Doctors at Duke University Hospital use the data to check how often antibiotics are given on time, and they report turning to this data to help decrease the amount of time operating rooms sit empty between cases.

But getting buy-in from some hospitals has been difficult, because there haven’t yet been any large, peer-reviewed studies showing how black boxes actually help to reduce patient complications and save lives. Mount Sinai’s chief of general surgery, Celia Divino, says that too much data can be paralyzing. “How do you interpret it? What do you do with it?” she asks. “This is always a disease.”

Read the full story by Simar Bajaj here.