notebooks/elasticBLAST/run_elastic_blast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c3f3bb2",
   "metadata": {},
   "source": [
    "# Run ElasticBLAST on GCP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aee3b229",
   "metadata": {},
   "source": [
    "This tutorial is a notebook version of [this ncbi tutorial](https://blast.ncbi.nlm.nih.gov/doc/elastic-blast/quickstart-gcp.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "This notebook helps you to run Blast in a scalable manner using Google Kubernetes. The script will spin up and later tear down a Kubernetes cluster. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "If at any point, you get an API has not been enabled error, go to [this page](https://cloud.google.com/endpoints/docs/openapi/enable-api#console), click `Go to APIs and Services`, then search for you API and click `Enable`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "+ Learn to use Kubernetes to scale BLAST jobs.\n",
    "+ Learn how to use BLAST in the cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38dfb579",
   "metadata": {},
   "source": [
    "### Install packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d96bb988",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip3 install elastic-blast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2aa11ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "! elastic-blast --version\n",
    "! elastic-blast --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b59cb0",
   "metadata": {},
   "source": [
    "### Optionally, create a bucket for this tutorial if one does not yet exist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "319ff226",
   "metadata": {},
   "outputs": [],
   "source": [
    "! gsutil ls gs://elasticblast-${USER} >& /dev/null || gsutil mb gs://elasticblast-${USER}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38267c47-029c-4026-8dc1-6020f978e496",
   "metadata": {},
   "outputs": [],
   "source": [
    "! gsutil ls gs://elasticblast-jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449d7511",
   "metadata": {},
   "source": [
    "### Create a config file that defines the job parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0017943-bbd3-472f-a3f8-30d88777f70a",
   "metadata": {},
   "source": [
    "Confirm your user name to include in the config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4bd7452-79ea-4c8b-a13e-b46cff6a5564",
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo ${USER}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b578c1ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "! touch BDQA.ini"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b0a866",
   "metadata": {},
   "source": [
    "Open the config file and add the following:\n",
    "```\n",
    "[cloud-provider]\n",
    "gcp-project = YOUR_GCP_PROJECT_ID\n",
    "gcp-region = us-east4\n",
    "gcp-zone = us-east4-c\n",
    "\n",
    "[cluster]\n",
    "num-nodes = 6\n",
    "num-cpus = 30\n",
    "labels = owner=jupyter\n",
    "\n",
    "[blast]\n",
    "program = blastp\n",
    "db = refseq_protein\n",
    "queries = gs://elastic-blast-samples/queries/protein/BDQA01.1.fsa_aa\n",
    "results = gs://elasticblast-jupyter/results/BDQA\n",
    "options = -task blastp-fast -evalue 0.01 -outfmt \"7 std sskingdoms ssciname\"\n",
    "```\n",
    "Replace _YOUR_GCP_PROJECT_ID_ with your actual project ID. The default CPUs for the cluster is 16 CPUs, here we set it to 30 to allow enough CPUs per job.\n",
    "\n",
    "You can add additional configuration values from [this guide](https://blast.ncbi.nlm.nih.gov/doc/elastic-blast/configuration.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9f8192",
   "metadata": {},
   "source": [
    "### Submit the BLAST job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "398253e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "! elastic-blast submit --cfg BDQA.ini"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8e7716",
   "metadata": {},
   "source": [
    "### Check results and troubleshoot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a43c5e",
   "metadata": {},
   "source": [
    "It should take a about 15-20 min to spin up your cluster and start submitting jobs. You can check the status of your job by opening a terminal within this instance, and paste in `elastic-blast status --cfg BDQA.ini`. You can also go to Kubernetes Engine and monitor the health of your cluster, or interact with the pods via `kubectl`. For example, in the terminal you can type `kubectl get pods`, to see your pods, then use `kubectl describe pods <pod-name>` to get details of a particular pod, and use `kubectl logs <pod-name>` to view the status of a particular pod. You can also monitor the cloud bucket with `!gsutil ls gs://elasticblast-jupyter/` to see if results files are being written."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "You just learned how to spin up a Kubernetes cluster and scale your BLAST job on Google Cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74ce84e2-43db-46b3-b3e5-af401587e28a",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3961d577-72b5-4a06-8597-7c724cf278c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "! elastic-blast delete --cfg BDQA.ini"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "common-cpu.m93",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m93"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}