In this field review, 2022 Just Tech Fellow Kim Gallon examines the rise of generative artificial intelligence in the healthcare space and how some researchers and machine learning advocates back its use to strengthen health datasets and make them more diverse. Gallon traces the history of health data inequality in the United States and explores the ethical and equity implications of using artificial health data in health research.