Citation

Repairing Innovation. A Study of Integrating AI in Clinical Care

Author:
Elish, Madeleine Clare; Watkins, Elizabeth Anne
Year:
2020

A team at Duke University and Duke Health system developed Sepsis Watch, an AI system that uses deep learning to assess a patients’ likelihood of developing sepsis, to support the Duke Emergency Department with caring for patients with sepsis. Sepsis is a deadly condition that develops from complications with an infection, and while treatable, it can be difficult to diagnose, and early diagnosis is critical.

Elish and Watkins chronicle the integration of Sepsis Watch through a sociotechnical lens: one that acknowledges the human labor required to harmonize a technical system with existing organizational and social structures. The integration of an AI system creates breakages in social structures that must be repaired in order for the technology to work as intended. Rapid Response Nurses at the Duke University hospital took on the bulk of this repair work, according to the authors; work that is often hidden and undervalued. For instance, they mediated professional hierarchies and performed emotional labor to strategically communicate patients’ risk scores to doctors.

Technological systems don’t exist in a bubble. They require a complex interaction of humans, infrastructure, and organizational structure to work effectively. Using a sociotechnical framework, and valuing the work of repair and those who do it, expands our notion of where innovation happens by valuing different forms of expertise and highlighting the importance of different types of workers.

Innovation happens not only when an AI system is being developed, but also when the system is being integrated. This is important to recognize because “when only the work of initiation and theoretical construction–typically elite and masculine forms of work–are valued in AI, then much of the actual day-to-day work required to make AI function in the world is rendered invisible and undervalued.”