Project idea: Improve annotation export for AI training

[petebankhead]

Background

A lot of questions on the forum are about exporting annotations and image patches in specific ways - usually for AI training.

Currently, the main approach is to use a scripting recipe from

• https://qupath.readthedocs.io/en/stable/docs/advanced/exporting_images.html
• https://qupath.readthedocs.io/en/stable/docs/advanced/exporting_annotations.html

Project

The challenge is that everyone seems to want export in some slightly different way... so the current recipes often aren't quite enough.

The goals of this project are to:

• Improve the scripting recipes, to cut down the number of necessary questions
• Develop a UI/wizard that guides a user through export

[zindy]

I like this. I was quite happy when I managed to export pre-processed tiles containing object instances, but then I noticed that my total number of instances was greater than 65535 and the tiles were exported as RGB images.

I have to admit, I wasn't able to modify the LabeledImageServer code to only consider total number of instances per tile.

[petebankhead]

Yes, beyond 65535 was problematic using Java2D to fill the pixels - since it's a bit limited in terms of bit-depths and types.

However, as far as I recall, I used the RGB image as a trick to overcome the limit, because Java uses a packed int representation for RGB. So it effectively raises the limit for distinct labels to 24-bit (I think I avoided using alpha).

Basically, the RGB pixels shouldn't be interpreted as 3 channels but rather 3 bytes that together can be joined to form a single int. That can be efficient, because me to create a Color object that way and paint it directly to the image - it just won't let me save it as a 24/32-bit integer image, but rather only as an RGB one.

So basically QuPath should be preserving all the labels properly, and the challenge is for any consumers to sort it out on the other side by 're-packing' the bytes to get the required integers.

Or, alternatively, just treating colors as labels - every unique label should have a corresponding unique color, even if you don't bother to pack the RGB values to find out what the label was meant to be.

The only other way I could think of doing it was to creating a separate 8-bit mask for every label, and then looping through all the pixels in the mask and setting the corresponding label in a float raster - but I'd expect that to be far more computationally expensive. It's also not possible (as far as I know) to write 32-bit PNGs that aren't RGB, so it would restrict the available file formats that could be used. Therefore I quite liked my 32-bit trick*....

*-Caveat, I haven't checked it works for a long time, and I might be misremembering it all.

[SalmaDammak]

I'd really like to be able to specify the image downsampling method when resampling images as part of the TileExporter call but I cant find where the default downsampling algorithm is specified, if someone knows where it is, that would be tremendesouly helpful, but otherwise I'll keep digging...

[MichaelSNelson]

I have not looked it up in the code (so @petebankhead please correct me), but I am fairly confident the only type of downsampling that would be supported by default would be a mean. Other methods (decimation, median, etc.) would not preserve the quantitative nature of the data, and that is usually the point of this type of analysis (not always, but usually). Other methods tend to be useful for upsampling - there is a whole variety of options, but that is creating data rather than preserving it.

[SalmaDammak]

@MichaelSNelson yes you're right, but my interest is more in controlling the up-sampling algorithm. From an offline discussion with @petebankhead it seems like the easiest method to do this is to just do it external to QuPath after exporting the tiles in their original resolution. Since I want to do this for slides with a variety of resolutions while maintaining the same physical output size (i.e. in microns) for tiles, I put together a version of the standard tile exporting script that does this.

[petebankhead]

@SalmaDammak Some hours later I realised that you could do the upsampling in QuPath in a script by calling ImageJ or OpenCV, although then you'd need to bypass the TileExporter.

Let me know if you'd strongly prefer to do this for any reason. Otherwise, I think your existing script is the better approach anyway: smaller export size, more upsampling options elsewhere.

The one thing I think it's missing is encoding the pixel size in the output - although I guess you can calculate that accurately (with just a bit of rounding error) from the known image size in µm anyway. If you did want the pixel size stored more directly, I can think of three ways to do it:

• export a text/json file containing this metadata and store it alone with the PNG images
• save as ImageJ TIFF (but that'll be uncompressed)
• save as OME-TIFF, with lossless compression

Both of the last two options might require adjusting some of the TileExporter to squeeze the pixel size into the image, or script around the need for the TileExporter altogether - so might not be worth it if it doesn't really matter to you.

[SalmaDammak]

Do you have an example script of how I can do the export with by calling ImageJ or OpenCV instead of using tile exporter? I don't strongly prefer it per say, but I'm curious how I'd do that and it sounds like it'd give me more exporting flexibility than the tile exporter.

Good catch on the pixel size, I'd rather have the exact value than recalculate and I actually prefer the first solution from what you proposed. How can I do that?

[petebankhead]

I've only time to give a quick answer to the first question now, but basically you'd use

• IJTools.convertToImagePlus(ImageServer, RegionRequest).getImage() to get an ImageJ-friendly image.
• OpenCVTools.imageToMat(BufferedImage) to get an OpenCV-friendly Mat

then it's a matter of grappling with their respective API's (which unfortunately aren't entirely intuitive, at least in Java).

For the export, writing to a text file in Groovy is especially straightforward, so you could do something like:

def cal = getCurrentServer().getPixelCalibration()
def file = new File('/path/to/file.txt')
file.text = cal.pixelWidth + ", " + cal.pixelHeight

If you want a single JSON file, then

def cal = getCurrentServer().getPixelCalibration()
def map = [
  "name": getProjectEntry().getImageName(),
  "pixel_width": cal.pixelWidth,
  "pixel_height": cal.pixelHeight,
]
// Get Gson (the true is for prettier printing)
def gson = GsonTools.getInstance(true)

def file = new File('/path/to/file.json')
file.text = gson.toJson(map)

That will just give you the pixel size for one image. You could export the pixel size for all images in a project in a single file with something like this:

def myList = []
for (def entry in getProject().getImageList()) {
    // Create the ImageServer using try-with-resources to ensure 
    // it's closed even if something goes wrong
    try (def server = entry.getServerBuilder().build()) {
        def cal = server.getPixelCalibration()
        def map = [
          "name": entry.getImageName(),
          "pixel_width": cal.pixelWidth,
          "pixel_height": cal.pixelHeight,
        ]
        myList << map
    }
}

def gson = GsonTools.getInstance(true)
// Uncomment & enter path to write to a file...
// new File(path).text = gson.toJson(myList)
println gson.toJson(myList)

[SalmaDammak]

This is super helpful, thanks @petebankhead!!

[petebankhead]

@SalmaDammak here's the script from this afternoon (or morning) :)

// Delete old annotations created by this script
// (to make debugging easier when running multiple times...)
def toDelete = getAnnotationObjects().findAll(p -> p.getPathClass() == getPathClass('My new intersection'))
removeObjects(toDelete, true)

// Get all annotations classified as tumor
def hierarchy = getCurrentHierarchy()
def annotations = getAnnotationObjects()
def tumorAnnotations = annotations.findAll(a -> a.getPathClass() == getPathClass('Tumor'))

// Store list of new annotations
// Means we can add them in one go (and avoid firing lots of addObject events)
def newAnnotations = []

// Loop through tumor annotations
for (def a in tumorAnnotations) {
    // Get all the objects that overlap with the ROI bounding box
    def region = ImageRegion.createInstance(a.getROI())
    def overlapping = hierarchy.getObjectsForRegion(null, region, null)
    overlapping.remove(a)
    
    // Check for objects that overlap the tumor annotation
    def geometry = a.getROI().getGeometry()
    overlapping = overlapping.findAll(a2 -> a2.isAnnotation() && geometry.intersects(a2.getROI().getGeometry()))
    
    // Loop through the overlapping objects
    for (def overlappingObject in overlapping) {
        // Compute the intersection
        def overlappingROI = overlappingObject.getROI()        
        def roiIntersection = RoiTools.intersection([a.getROI(), overlappingROI])
        
        // Calculate dice score
        def areaIntersection = roiIntersection.getArea()
        def dice = 2 * areaIntersection / (overlappingROI.getArea() + a.getROI().getArea())
        
        // Create new annotation for the intersection and add dice score as a measurement
        def newAnnotation = PathObjects.createAnnotationObject(roiIntersection, getPathClass('My new intersection'))        
        
        newAnnotation.getMeasurementList().putMeasurement('Dice', dice)
        newAnnotation.getMeasurementList().close()        
        newAnnotations << newAnnotation
    }
    
}

// Add the new (intersection) annotations all in one go
addObjects(newAnnotations)

[SalmaDammak]

Thank you so much!!