Resource optimisation algorithm

[priyanka-surana]

The project is to create an optimisation algorithm for Nextflow pipelines, using input data size and resources used by software in previous run.

[cibinsb]

The problem statement revolves around developing a computational resource optimization algorithm for effectively running pipelines on a High-Performance Computing (HPC) cluster. The primary objective is to design an algorithm that can accurately determine the required resources, such as CPU and memory, based on historical data extracted from the pipeline execution logs. These logs contain valuable information regarding the input sizes and the corresponding resources utilized by pipelines in their previous runs.

To address this challenge, the algorithm needs to leverage historical data to establish patterns and correlations between input sizes and resource consumption. By analyzing past runs, the algorithm can identify trends, optimize resource allocation, and ensure efficient utilization of the HPC cluster. It should be capable of dynamically adapting the allocation based on the characteristics of the pipelines, enabling the cluster to handle varying workloads effectively.

By employing statistical analysis and machine learning techniques, the algorithm can accurately predict the resource requirements for upcoming pipelines based on similar past cases. This predictive capability will enable proactive allocation, reducing the likelihood of resource shortages or wasteful overallocation.

Overall, the development of this algorithm aims to enhance the efficiency and productivity of running pipelines on the HPC cluster. By leveraging historical data from the logs, the algorithm can make informed resource allocation decisions, ensuring optimal utilization of CPU, memory, and other resources, while accommodating the varying demands of different pipelines.