Examine the unique challenges with medical data labeling, the relative lack of accuracy produced by traditional data labeling methods, and a more accurate and scalable alternative based on collective intelligence.

The Medical AI Transformation

Healthcare is undergoing a massive transformation through the adoption of medical AI. This is a trend that we expect to accelerate as more AI companies receive approval for reimbursements. (for more on that, check out this article from Luke Oakden-Rayner) 

‍

‍

AI is, of course, only as good as the data used to train the model. It takes a tremendous amount of highly accurate and meticulously labeled data to properly train the latest deep learning models. As such, the data labeling has ballooned into a multi-billion dollar industry, primarily by the explosive growth of data labeling for the autonomous vehicle, AR/VR, and retail industries

Medical data labeling requires skill

However, within healthcare, there are unique challenges with acquiring highly accurate medical training data. First and foremost, it is a task that requires skill. This means that traditional data labeling methods are unable to produce labels accurately enough to be useful for a medical AI application. 

Many practitioners have attempted to compensate for the skills gap by creating extensive training programs for their labelers or hiring teams of board certified physicians. These measures drive cost up and are still not sufficient in delivering accurate results since they rely solely on the credentials and education of the labelers rather than evaluating their recent performance on each specific data labeling task. 

To put this in a different perspective, if you were to ask a radiologist how good they are, they will say they are ‘good’. Digging deeper, if you ask them if they are better at finding calcifications or masses in breast x-rays, they might be able to give a subjective answer but they have no quantifiable way of indicating their performance relative to their peers. What’s more, it is becoming well known that medical experts disagree at alarming rates. In fact, a recent study by Cheng et al (2013) found that radiologists disagreed on 16% of CT scans at a level 1 trauma center. This isn’t a knock on physicians. Rather, it is the result of not having a common way to evaluate the labeling performance of experts on a given task. 

"Radiologists disagreed on 16% of CT scans at a level 1 trauma center" - Cheng et al., (2013)

‍

Challenges with medical data labeling

In addition to the inability to access labeler performance and reconcile disagreements, there are many other challenges with medical data labeling as show in the table below:

‍

‍

In our effort to better understand the challenges in medical data labeling, we interviewed dozens of experts in the medical AI and annotation space. We captured their insights into what we hope is a descriptive guide for anyone looking to enhance the accuracy and performance of their medical data labeling efforts.

From this, we created a free guide for anyone who is looking to improve their medical data labeling.

Download the white paper to improve your medical data labeling

‍

‍

‍

‍

You'll learn the following:

‍

Why medical data labeling is different

Learn the unique challenges of working with medical data including the high skill needed for labeling and managing privacy concerns of PHI

How to collect medical data

Explore ways to acquire medical data including open-source, in-house and through licensing and partnerships

How to clean and enrich medical data

Understand ways to clean, classify and segment medical data and when to employ each labeling method

Options for medical data labeling

Review data labeling vendor models including in-house experts, hiring medical students, hybrid teams and crowdsourced options

How to evaluate accuracy of medical data labels

Grasp how to evaluate the accuracy of your medical data labels and understand where traditional methods fall short

The benefits of collective intelligence

Discover a new method for data labeling that offers superior accuracy vs other methods by aggregating multiple opinions