interactive_MS_analysis_replicates_WORKSTATION.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mass Spectrometry Analysis Notebook\n",
    "\n",
    "## Purpose\n",
    "This notebook is designed to facilitate the analysis of mass spectrometry (MS) data for triplicate experiments. It supports:\n",
    "- Loading and preprocessing MS data.\n",
    "- Annotating proteins of interest (POIs) based on a user-provided database.\n",
    "- Mapping proteins to specific classes or families with color codes.\n",
    "- Generating volcano plots to visualize differential abundance.\n",
    "- **Search** the plot for specific proteins \n",
    "- Creating heatmaps for selected protein classes or families.\n",
    "\n",
    "## Input File Formats\n",
    "To successfully run this notebook, you must provide the following files in the specified formats:\n",
    "\n",
    "### 1. **Mass Spec Data File (`ms_data_path`)**\n",
    "- **File Type**: Tab-delimited `.txt`\n",
    "- **Contents**: Quantitative MS data with a column labeled `Accession` (for protein identifiers) and abundance columns prefixed by `apQuant Area`.\n",
    "\n",
    "### 2. **Proteins of Interest (POI) File (`poi_file_path`)**\n",
    "- **File Type**: Excel `.xlsx`\n",
    "- **Contents**: A database of proteins of interest (POIs) containing:\n",
    "  - A column named `Other UniProt Accessions` listing protein accessions.\n",
    "  - A column named `Class / family` specifying the classification/group of the proteins.\n",
    "\n",
    "### 3. **Color Code File (`color_code_file_path`)**\n",
    "- **File Type**: CSV\n",
    "- **Contents**: Class-specific color codes, containing:\n",
    "  - A column named `Class / family` corresponding to the POI file.\n",
    "  - A column named `Color Hex` with hexadecimal color codes for visualization.\n",
    "\n",
    "## Configurable Parameters\n",
    "The following settings allow users to customize plot appearance and analysis behavior:\n",
    "\n",
    "### Volcano Plot Settings\n",
    "- **Plot Dimensions**:\n",
    "  - `plot_width`: Width of the plot (default: 800).\n",
    "  - `plot_height`: Height of the plot (default: 600).\n",
    "- **Thresholds**:\n",
    "  - `show_threshold_lines`: Show or hide threshold lines (default: `True`).\n",
    "  - `log2_fc_positive`: Positive log2 fold-change threshold (default: `2`).\n",
    "  - `log2_fc_negative`: Negative log2 fold-change threshold (default: `-2`).\n",
    "  - `p_value_threshold`: P-value threshold (default: `0.05`).\n",
    "- **Dot Appearance**:\n",
    "  - `unlabeled_dot_size`: Dot size for unlabeled proteins (default: `5`).\n",
    "  - `labeled_dot_size`: Dot size for labeled proteins (default: `10`).\n",
    "  - `unlabeled_opacity`: Opacity for unlabeled protein dots (default: `0.5`).\n",
    "  - `labeled_opacity`: Opacity for labeled protein dots (default: `1`).\n",
    "\n",
    "### Directory for Outputs\n",
    "- **Output Directory**: `output_directory` (default: `./plots`)\n",
    "\n",
    ">Note:  In the code block below, Windows paths must be formatted as absolute paths like this `path = r\"/path/to/your/file\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of proteins in the mass spec file: 768\n",
      "Number of POIs found in the mass spec file: 81\n",
      "Number of proteins in the POI file: 572\n",
      "Number of Classes / families in the POI file: 18\n",
      "Number of Classes / families found in the mass spec file: 18\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Paths for tour data and annotation files\n",
    "ms_data_path = r\"\\\\storage.imp.ac.at\\groups\\plaschka\\shared\\data\\mass-spec\\spliceosome\\volcano_20230630_E3_RSLC6_Vorlaender_Plaschka_IMP_ID1242_onbead_dig_20per_6x_step2_normsum_limma_Proteins.txt\"  # Mass spec data from facility in .txt file\n",
    "poi_file_path = r\"\\\\storage.imp.ac.at\\groups\\plaschka\\shared\\data\\mass-spec\\MS_analysis\\protein_annotations\\POIs_c_elegens_spliceosome_database_entries.xlsx\"  # POI file containign the Uniprot Accessions of your proteins of interest as well as a column labelled \"Class / family\" that is used to identify groups of proteins)\n",
    "color_code_file_path = r\"\\\\storage.imp.ac.at\\groups\\plaschka\\shared\\data\\mass-spec\\MS_analysis\\protein_annotations\\COLORS_PolII_Spliceosome_250225_MV.csv\" #Color code file that contains the same Class / family  column as the POI file and a column labelled \"Color\" that contains the color codes for the proteins of interest\n",
    "\n",
    "# User-configurable settings for VOLCANO PLOT appearance\n",
    "plot_width = 800  # Width of the plot\n",
    "plot_height = 600  # Height of the plot\n",
    "\n",
    "# Threshold line settings\n",
    "show_threshold_lines = True  # Set to False to hide threshold lines\n",
    "log2_fc_positive = 4  # Position of the positive threshold line\n",
    "log2_fc_negative = -2  # Position of the negative threshold line\n",
    "p_value_threshold = 0.05  # P-value threshold\n",
    "\n",
    "# Dot size and opacity settings\n",
    "unlabeled_dot_size = 5  # Size of dots for unlabeled proteins\n",
    "labeled_dot_size = 10  # Size of dots for labeled proteins\n",
    "unlabeled_opacity = 0.5  # Opacity of dots for unlabeled proteins\n",
    "labeled_opacity = 1  # Opacity of dots for labeled proteins\n",
    "\n",
    "\n",
    "#Direcoctory in which outputs will be saved\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "output_directory = Path(r\"\\\\storage.imp.ac.at\\groups\\plaschka\\shared\\data\\mass-spec\\MS_analysis\\MV_ce_ILS_release\")\n",
    "\n",
    "#set the prefix of the data columns that contain protein abundance (NOT RECOMMENDED TO CHANGE)\n",
    "prefix = \"apQuant Area\"\n",
    "\n",
    "# Initialize and load data\n",
    "sys.path.append(r\"\\\\storage.imp.ac.at\\groups\\plaschka\\shared\\software\\scripts\\python\\ms_analysis_triplicates_src\")\n",
    "from ms_analysis_triplicates_utils import DataLoader\n",
    "\n",
    "data_loader = DataLoader(ms_data_path, poi_file_path, color_code_file_path, prefix)\n",
    "merged_data = data_loader.load_and_process_data(debug=False)  # This will set color_mapping_dict\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Volcano plot\n",
    "The following cell sets up an interactive interface for generating volcano plots. Dropdown menus allow you to select the columns for **Log2 Fold Change** (`log2 FC`) and **p-values**.\n",
    "\n",
    "#### Adjusting Keywords for Column Filtering ####\n",
    "\n",
    "If the drop-down menus for selecting **Log2 FC Columns** or **p-value Columns** do not show the expected options, you can look for these lines in the code vell below and modify the filtering keywords:\n",
    "\n",
    "1. **Understand the Filtering Logic**:\n",
    "  \n",
    "   The variables below define the keywords used to identify relevant columns in the `merged_data` DataFrame. \n",
    "   Note that files from the MS facility don't always have the exact same naming, so you moght need to adjust the keywords, but try the default first\n",
    "   ```python\n",
    "   log2_fc_keywords = ['log2']  # Keywords for identifying Log2 FC columns\n",
    "   pval_keywords = ['pval', 'p-value']  # Keywords for identifying p-value columns\n",
    "   required_keyword = 'imputed'  # Common keyword required in all selected columns\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6edec190beee49f2bd98375877be514b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Dropdown(description='Log2 FC Column:', layout=Layout(width='400px'), options=('log2Fold change…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57eef1addd004a8a8f6c51912063b3b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), VBox(children=(VBox(children=(HBox(children=(Text(value='volcano_plot', description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "import sys\n",
    "import pandas as pd\n",
    "from ipywidgets import Dropdown, VBox, Output, Button, Layout\n",
    "from IPython.display import display\n",
    "import plotly.graph_objects as go  # Added import\n",
    "\n",
    "# Define keywords for identifying the columns for plotting\n",
    "log2_fc_keywords = ['log2']  # Keywords for log2 FC columns\n",
    "pval_keywords = ['pval', 'p-value']  # Keywords for p-value columns\n",
    "required_keyword = 'imputed'  # Common required keyword for both\n",
    "\n",
    "\n",
    "from ms_analysis_triplicates_utils_dev import VolcanoPlotManager, InteractiveFieldSelector\n",
    "\n",
    "plot_manager = VolcanoPlotManager(\n",
    "    plot_width=plot_width,\n",
    "    plot_height=plot_height,\n",
    "    show_threshold_lines=show_threshold_lines,\n",
    "    log2_fc_positive=log2_fc_positive,\n",
    "    log2_fc_negative=log2_fc_negative,\n",
    "    p_value_threshold=p_value_threshold,\n",
    "    unlabeled_dot_size=unlabeled_dot_size,\n",
    "    labeled_dot_size=labeled_dot_size,\n",
    "    unlabeled_opacity=unlabeled_opacity,\n",
    "    labeled_opacity=labeled_opacity,\n",
    "    color_mapping_dict=data_loader.color_mapping_dict,\n",
    "    output_directory=output_directory,\n",
    "    debug=False\n",
    ")\n",
    "\n",
    "\n",
    "\n",
    "# 2) Use InteractiveFieldSelector for hover fields\n",
    "field_selector = InteractiveFieldSelector(\n",
    "    data=merged_data,\n",
    "    default_hover_fields=[\"Accession\", \"Description\"]\n",
    ")\n",
    "plot_manager.set_field_selector(field_selector)\n",
    "\n",
    "# 3) Filter columns using keywords\n",
    "log2_fc_cols = [\n",
    "    col for col in merged_data.columns \n",
    "    if required_keyword in col.lower() and any(k in col.lower() for k in log2_fc_keywords)\n",
    "]\n",
    "pval_cols = [\n",
    "    col for col in merged_data.columns \n",
    "    if required_keyword in col.lower() and any(k in col.lower() for k in pval_keywords)\n",
    "]\n",
    "\n",
    "# 4) Adjust dropdown widgets\n",
    "fc_dropdown = Dropdown(\n",
    "    options=log2_fc_cols,\n",
    "    description=\"Log2 FC Column:\",\n",
    "    layout=Layout(width=\"400px\"),\n",
    "    style={'description_width': '200px'}\n",
    ")\n",
    "pval_dropdown = Dropdown(\n",
    "    options=pval_cols,\n",
    "    description=\"p-Value Column:\",\n",
    "    layout=Layout(width=\"400px\"),\n",
    "    style={'description_width': '200px'}\n",
    ")\n",
    "\n",
    "plot_output = Output()\n",
    "\n",
    "# 5) Button to generate the volcano plot\n",
    "def generate_plot(_):\n",
    "    with plot_output:\n",
    "        plot_output.clear_output()\n",
    "        fc_col, pv_col = fc_dropdown.value, pval_dropdown.value\n",
    "        if fc_col and pv_col:\n",
    "            plot_manager.result = merged_data\n",
    "            fig = plot_manager.create_volcano_plot(\n",
    "                data=merged_data,\n",
    "                log2fc_col=fc_col,\n",
    "                pval_col=pv_col,\n",
    "                title=f\"Volcano Plot: {fc_col} vs {pv_col}\",\n",
    "                hover_fields=field_selector.get_selected_hover_fields()\n",
    "            )\n",
    "            plot_manager.current_figure = fig\n",
    "            #lot_manager._update_layer_checkboxes()\n",
    "            plot_manager._update_static_label_dropdown()\n",
    "            print(\"Plot generated successfully!\")\n",
    "\n",
    "generate_button = Button(description=\"Generate Plot\", button_style=\"success\")\n",
    "generate_button.on_click(generate_plot)\n",
    "\n",
    "# 6) Simple UI for generating plot\n",
    "ui = VBox([\n",
    "    fc_dropdown,\n",
    "    pval_dropdown,\n",
    "    field_selector.hover_box,\n",
    "    generate_button,\n",
    "    plot_output\n",
    "])\n",
    "display(ui)\n",
    "\n",
    "# 7) Finally, display the advanced manager interface (which includes the search buttons)\n",
    "plot_manager.display_interface()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Heatmap plots\n",
    "\n",
    "The next cells initializes the `HeatmapManager` and sets up an interactive interface for generating heatmaps. Here's what happens in this cell:\n",
    "\n",
    "\n",
    "1. **Interactive Field Selector**:\n",
    "   - The `InteractiveFieldSelector` is used to select additional fields for hover text in the heatmaps (e.g., `Accession` and `Description`).\n",
    "\n",
    "2. **Displaying the Interface**:\n",
    "   - The interface allows you to:\n",
    "     - Select coloring methods for the heatmaps (`auto` or `manual`).\n",
    "     - Choose specific classes, label fields, and log2 fold-change columns for plotting.\n",
    "     - Generate heatmaps and save them interactively.\n",
    "     - One plot will be generated for each selected Class/ family with the option to save as hrmla nd svg\n",
    "\n",
    "Run this cell to activate the interface and start customizing your heatmap generation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7875d33e705e4eca9b4e5ccb094135bb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(VBox(children=(Dropdown(description='Heatmap coloring method:', layout=Layout(width='450px'), o…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Initialize HeatmapManager\n",
    "from ms_analysis_triplicates_utils import (\n",
    "    HeatmapManager,\n",
    "    InteractiveFieldSelector\n",
    ")\n",
    "heatmap_manager = HeatmapManager(result=merged_data, output_directory=output_directory / \"heatmaps\" )  # Use pathlib for path joining\n",
    "\n",
    "# 4) Use InteractiveFieldSelector for hover fields\n",
    "field_selector = InteractiveFieldSelector(\n",
    "    data=merged_data,\n",
    "    default_hover_fields=[\"Accession\", \"Description\"]\n",
    ")\n",
    "# Set the field selector\n",
    "heatmap_manager.set_field_selector(field_selector)\n",
    "\n",
    "# Display the interface\n",
    "heatmap_manager.display_interface()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (ms_analysis)",
   "language": "python",
   "name": "ms_analysis"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.21"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}