Beware of public datasets in healthcare.

@April 28, 2022

As interest in data science and machine learning in healthcare as exploded, it's become popular to release large, open datasets. Some famous examples of this are the canonical critical care EMR dataset MIMIC, the chest x-ray dataset CXR-14, and EyePACS dataset for eye fundus images.

As these datasets have grown in renown, the incentive to gather and release public datasets for the accompanying publications and fame has grown. The positive side of this is that the number of datasets in healthcare has increased tremendously. We're now at a point where there are meta-analyses of healthcare datasets, like this one in the Lancet on ophthalmology datasets.

The downside, however, is that the amount of bad data has skyrocketed. Because of the emphasis on releasing large amounts of data, without agreements on standards or with the resources to support such datasets for the long run, there is now a proliferation of large, noisy datasets.

Luke Oakden-Rayner published a great analysis of the Chest X-Ray14 dataset at the time it was released, with clear explanations for the noisiness of the dataset. I myself have had many experiences with public datasets that seem promising at first, but hopelessly noisy upon further examination.

I worked with an optometrist to evaluate the Kermany dataset, which is the largest and most commonly used (~30K downloads) public OCT dataset. We found that the dataset had a disproportionate number of myopic eyes, too much imaging device variation, and quality control issues with image processing (intensity averaging and field of view cut offs). All of this combined to make the dataset useless for our medical AI application of diagnosing retinal disease in the real-world.

It’s important to note that bad public datasets are not just healthcare-specific. Northcutt and Athalye of MIT found common machine learning benchmark datasets like ImageNet were of poor quality, with “label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets”. Galileo finds tons of errors in NLP datasets. Hasty.ai found that cleaning up the PASCAL computer vision dataset increased modeling performance by 13% (!), a massive improvement.

Bad data is rampant in healthcare and everywhere.

This negative results of all this bad data proliferating are several-fold.

It's hard to build anything meaningful with public data. With the avalanche of poor quality datasets only increasing, it's getting harder. Rather than being empowered by the vast amount of data, the data scientist or ML engineer wastes a lot of time parsing and characterizing the noise in the dataset.

It diminishes trust in medical applications of AI, because systems trained on large amounts of public data by non-healthcare people generalize poorly. This is frustrating both for developers, who find the problem area to be more obtuse than initially expected, and customers, who despise the accompanying hype cycle.

It perpetuates a broken cycle for medical AI maturity, which acts like an ImageNet moment is just waiting around the corner. The reality is that medical AI depends heavily on the context of the application of the algorithm, not so much on the quantity of the data used to train a model. Rather than spending time developing huge datasets, it would be better to get the community focused on standards for AI deployment, as well as open standards for model training. In a nutshell, more data has never been the problem for healthcare AI; people just assumed it was because it was the problem in other areas (i.e. image recognition).

All of this begs the question of how to actually put together a good healthcare dataset. Here are some ideas for how to release higher quality public datasets: