ImageAnalysis

Automated Image Analysis

Introduction

Images stored in Breedbase can be subjected to automated analyses using a standardized protocol and store the results, usually keyed to a trait, into the database.

User interface

The user interface to submit images to analysis is available on Breedbase at the URL /tools/image_analysis.

Images can be queried and selected from a list

Then users can pick an analysis service to submit them to

The analysis services are currently defined in mason/tools/image_analysis.mas. This should be refactored into an entry in sgn_local.conf.

Analysis progress is reported

Then results are displayed for review, with the option to save the scores as new phenotypes.

Backend implementation

JSON call format

The structure of the call is defined as follows:

my $resp = $ua->post(
      "http://unet.mcrops.org/api",
      Content_Type => 'form-data',
      Content => [
          image => [
            '/home/production/volume/public/images/image_files_test/5b/73/32/47/f52d22ee4227e1c9049ef4b8/image-3.png',
            Content_Type => 'image/png'
          ],
      ]
    );

JSON return format

{
               'image_name' : 'Sr9GHXyXCN_nrVV13r.JPG',
               'trait_value' : '33.75',
               'image_link' : 'http://unet.mcrops.org/api_results/Sr9GHXyXCN_nrVV13r.png',
               'trait_name' : 'CBSDpct | CO_334:0002078',
               'analysis_info': { 
                    // any keys and values describing the analysis parameters
               }
};

Here is an exmaple response from Makere University's Necrosis analysis service which works with Breedbase. It includes the necessary fields in addition to many more.

{
    "results": [
        {
            "stock_uniquename": "C2_NaCRRI_CET_GWAS_2019_1_1_Root1_image22",
            "image_description": null,
            "stock_type_name": "tissue_sample",
            "project_id": 6492,
            "image_original_filename": "C2_NaCRRI_CET_GWAS_2019_1_1_Root1_image2_1_2020-10-12-10-21-25",
            "observations_array": [],
            "result": {
                "value": "0.00",
                "trait_id": 78811,
                "analyzed_image_id": 58881,
                "trait": "CBSDpct|CO_334:0002078",
                "original_image": "https://cassavabase.org/data/images/image_files/be/ab/ee/59/740a92d844ea09cc95bf54bb/C2_NaCRRI_CET_GWAS_2019_1_1_Root1_image2_1_2020-10-12-10-21-25.jpg",
                "analysis_info": {
                    "ref-table": "https://virologyj.biomedcentral.com/articles/10.1186/s12985-014-0216-x",
                    "publications": "https://arxiv.org/abs/2005.03367",
                    "model-version": "v.3.1",
                    "data-access": "https://doi.org/10.1016/j.dib.2020.106170",
                    "parameters": {
                        "epochs": "217",
                        "images": "3428",
                        "batch-size": "4"
                    },
                    "info": "Model version 3.1 trained in 12/2022. Image input size changed from 128 to 256 pixels",
                    "architecture": "Unet-256",
                    "results": {
                        "trained": "12/2022",
                        "IOU": "0.931",
                        "Dice": "0.964"
                    }
                },
                "image_link": "/data/images/image_files/71/a3/16/65/6357e28131425ad000909af8/imageDnOT.png"
            },
            "project_image_type_name": "phenotype_spreadsheet_associated_images",
            "image_obsolete": 0,
            "image_file_ext": ".jpg",
            "project_md_image_id": 8899,
            "image_name": null,
            "image_md5sum": "beabee59740a92d844ea09cc95bf54bb",
            "image_modified_date": "2020-10-17T08:32:05-04:00",
            "image_id": 13164,
            "tags_array": [
                {
                    "create_date": "2019-10-07T07:51:53.630545-04:00",
                    "modified_date": "2019-10-07T07:51:53.630545-04:00",
                    "tag_id": 4,
                    "sp_person_id": 420,
                    "obsolete": false,
                    "description": "Upload phenotype spreadsheet with associated images: phenotype_spreadsheet_associated_images",
                    "name": "phenotype_spreadsheet_associated_images"
                }
            ],
            "image_username": "michealkanaabi",
            "stock_id": 2386839,
            "image_create_date": "2020-10-17T08:32:05-04:00",
            "image_sp_person_id": 1137,
            "project_name": "2019_C2_CET_GWAS_Namulonge"
        }
    ],
    "success": 1
}

Multi-object images (Data structure proposal)

Sometimes images contain multiple objects, such as multiple fruits, tissue extracts, roots, etc. that should be analyzed separately.

As a discussion starting point, the workflow could be implemented with the submission of one image, but several images could be returned, along with several scores, as well as a composite score with standard errors; for each returned image, a new tissue sample would have to be created on the fly that is associated with the current observation unit. (Of course other workflows are possible).

For the RESTful call, the current observation unit name should also be added as a parameter.

{
               'image_name' : 'Sr9GHXyXCN_nrVV13r.JPG',
               'trait_name' :  'CBSDpct | CO_334:0002078',
               'image_link' :  'http://unet.mcrops.org/api_results/Sr9asdfasdf3r.png',   # The composite of the analyzed image showing identified objects
               
               results : {
                  sample_1 : {
                     'image_link' : 'http://unet.mcrops.org/api_results/Sr9GHXyXCN_nrVV13r.png',  # the masked image with the analysis marked
                     'trait_value' : '32.10',
                   },

                   sample_2 : {
                      ...
                   },
                },

};