Overview of Domain (new)

Introduction

The core of the project is to make the HPC democratisable for Imaging researchers at WEHI. The project aims to propose a user-friendly interface that Imaging researchers can use to submit various workflows to the Milton HPC. The Bioimaging Team has attempted to run their scripts on Jupyter Notebook, Nextflow Tower and Python Flask. Python flask was initially the preferred software but unfortunately it is incompatible with Milton. As such, R/Shiny is now the preferred software as it is compatible with Milton and it serves as a good middle ground between Jupyter Notebook and Nextflow Tower.

Imaging Workflows

Imaging researchers have various workflows for different projects. A workflow typically consists of a script that will take an image and process it, returning the processed image or another type of file. The difference in workflows is that some require additional input from the researchers, as they need to customise the parameters for the script before the workflow is submitted to the HPC for processing. Other workflows do not require additional inputs. As such, the interface needs to be dynamic. It will need to handle different inputs for different workflows, based on the customisable parameters that each workflow contains.

To read more about the requests from the Imaging researchers, click here

App Interface

The interface that researchers will access must be simply an easy-to-use. The interface ideally should have a dropdown menu that allows the researchers to select which workflow they want to use. After the researcher clicks on a workflow, the interface changes dynamically based on the requirements of the workflow. Some workflows require input for the parameters, others do not. However, most workflows will require an input directory for the unprocessed image and an output directory for the processed image/data.

R/Shiny App

The Shiny App is a library provided by RStudio, it is a platform that allows a web interface to be launched and accessed by a local host. However, we will be focused on launching a Shiny App on OnDemand. In this case, we have 2 ways to launch the web interface: RStudio or the Shiny App itself. For the final implementation, the app should be available on the Shiny App for researchers to access instead of RStudio to reduce complexities.

The current version of the Shiny app can be found here. To understand more on how to launch the R/Shiny app on OnDemand, head to the R/Shiny App Wiki

Allowing Researchers to Access the R/Shiny App

When you follow the steps to launch the app through the R/Shiny App Wiki, the app is located in your StorNext directory. Because of this, other users are not able to access the app's directory due to the nature of Stornext. It is suggested to read more on WEHI's file systems. In essence, WEHI has StorNext and VAST.

In order for researchers to use the interface, they will need to be able to access the app's directory itself. This is where VAST comes in, more specifically VAST Scratch. VAST is WEHI's high-performance storage and by having the R/Shiny app located in VAST Scratch, this allows you to grant access to different users to access the app. This is because unlike StorNext, VAST allows users to grant access to specific directories for other users to access. To read more on this, read about VAST and the technical notes for granting permissions.