.

Confirming AI Decisions: Utilising Indica Labs HALO v3.5 Validation to Strengthen OracleBio’s Classifier Validation Workflow
30 March 2023
By Mark Laurie

⚡ Article Synopsis: In this article, OracleBio Image Analysis Scientist, Caitlin Rutherford, highlights how Indica Labs HALO v3.5 validation workflow aids us in streamlining our internal algorithm validation process and explains how we would utilise this workflow in a study.

Algorithm validation is one of my personal favourite aspects of an image analysis study as it produces data to support how well you think your algorithm is performing. It’s always extremely satisfying when the validation confirms how accurate the analysis is.

This is particularly true for Artificial Intelligence (AI) based algorithms. With the advancements of AI propelling rapidly in recent years, many people are wary of the generated data outputs. How do we know that our algorithms are accurate? They may look good on select images, but how can we trust them on the whole image set (and beyond)? Have we only perfected them for the images we’ve been looking at in the software? Are we biased?

By formally validating algorithms, we can alleviate some of these concerns.

At the Client’s request, we can perform validation for cell detection algorithms (e.g., “Are we accurately quantifying the CD3+ cell population?”) as well as tissue classifier algorithms (e.g., “Are we accurately segmenting tumour tissue vs stroma tissue?”).

So how do we approach this?

If a study contains n=100 images, we begin by splitting these into a training set and a validation set (See Figure 1). As a general rule, we aim to separate the images into an 80:20 split (80% of the images in the training set and the other 20% in the validation set).

The ratio of training vs validation images is customisable and can therefore be adjusted on a per-study basis. When deciding on the split of images, it’s important to include images representative of the study in both the training and validation set.

A subset of images from the training set will initially be used to train the classifier until enough markups have been made across a selection of representative images. Then, a visual review of the classifier’s performance can be carried out on the other images (those not used for markups) within the training set.

The other n=20 images from the validation set will remain untouched until the classifier has been finalised. It’s important to ensure that the validation set images will not be used in any training and will be used only to validate the classifier.

[Above]: Figure 1 – Example of images split into training and validation sets which is the first step in validating any algorithm. This ensures that the images being used for validation are kept separate from those training the algorithm which prevents bias. As shown above, a common split is 80:20 to ensure there are sufficient examples in each set.

Once the classifier is finalised, the first (and arguably the most important) step with the validation set is for a pathologist to generate the ground truth annotations. This ensures the performance of the classifier is being directly compared to regions identified by an expert.

Running the classifier within these regions allows the validation metrics to be generated. Figure 2 below summarises these metrics within the context of a classifier segmenting tumour vs stroma.

[Above]: Figure 2

Once the classifier has been run in all the regions, the classification of true or false positive areas as well as false negative areas can be used to generate the metrics to quantify a classifier’s accuracy (see below for more details).

In the past, this has involved a lot of individual regions being processed manually and lots of number crunching for each region.

The new functionality of the Indica Labs HALO v 3.5 Validation workflow has helped in automating and simplifying some of the steps highlighted above and incorporating them into a single analysis interface (see Figure 3). This includes splitting up the image sets (training vs validation), adding classifier training annotations and pathologist ground truth annotations and running the analysis for the validation metrics.

As a result, we can now perform our classifier validation in one process, which dramatically reduces the processing times and also improves the interface with the client.

[Above]: Figure 3 – The classifier validation process.

The validation workflow also automatically provides validation metric data, such as the recall and precision score as well as the F1-Score. Recall demonstrates how well the algorithm is classifying positivity compared to the Pathologist Ground Truth. Conversely, Precision measures how accurately the algorithm is classifying true positivity compared to overall detected positivity. The F1 Score takes both precision and recall into account to give an overall value as to how accurately the classifier is performing.

Personally, my favourite feature of the HALO Validation Workflow is the overlay displayed once the validation has been run. It shows each region of interest (ROI) and the associated true positive, false positive, and false negative regions. It’s also customisable with different colours for ease when viewing.

This is particularly useful as it can get very visually confusing when there are multiple regions and multiple misclassifications, so having this option to toggle the overlay has aided in demonstrating the validation results to clients.

In Summary

The validation of algorithms is something OracleBio is continuously improving, as it’s crucial to have the ability to prove the decisions of our AI algorithms are in agreement with a pathologist’s decision.

Furthermore, digital pathology is such a dynamic field and image analysis processes, including validation workflows, are often refined and improved as the software becomes more advanced.

Classifier validation is complicated with lots of regions of interest and classifications, so it can quickly grow arms and legs; so the HALO v3.5 validation workflow has been a welcomed update.

Having everything centralised to one process has reduced processing times for each validation dramatically, as well as decreased time spent on data management. The overlay function and its customisability have enabled a much cleaner visualisation of how the classifier is performing against the validation set and have allowed us to more clearly involve clients in the classifier validation process throughout.

It’s safe to say that I’m looking forward to seeing what new features Indica Labs release that will continue to support and refine our workflow!

I hope you found this article useful – if you’d like to find out more about our workflow and processes or to discuss how OracleBio can support your image analysis studies, don’t hesitate to get in touch and arrange a chat with us.

James Going

About the author: 

Caitlin Rutherford – Image Analysis Scientist

.

How OracleBio can support you

Why work with OracleBio?

Stay up to date with OracleBio

Sign up to our mailing list to stay informed about news, webinars, events and more