reogranize

examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_image_data/image_stats/job_api/image_stats_job.py → examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_image_data/code/image_stats_job.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 import argparse
 
-from image_statistics import ImageStatistics
+from src.image_statistics import ImageStatistics
 
 from nvflare.job_config.stats_job import StatsJob
 

examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_image_data/federated_statistics_with_image_data.ipynb

@@ -31,7 +31,7 @@
    },
    "outputs": [],
    "source": [
-    "%pip install -r image_stats/requirements.txt"
+    "%pip install -r code/requirements.txt"
    ]
   },
   {
@@ -76,21 +76,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "bdc68ebf-6071-479d-8cc1-15439bedea02",
    "metadata": {
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "total 0\n",
-      "drwxr-xr-x@ 12 kevlu  staff  384 Jan 30 21:55 \u001b[34mCOVID-19_Radiography_Dataset\u001b[m\u001b[m/\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "ls -l /tmp/nvflare/image_stats/data/."
    ]
@@ -107,49 +98,59 @@
     "\n",
     "Next, create the data lists simulating different clients with varying amounts and types of images. \n",
     "The downloaded archive contains subfolders for four different classes: `COVID`, `Lung_Opacity`, `Normal`, and `Viral Pneumonia`.\n",
-    "Here we assume each class of image corresponds to a different sites.\n",
-    "\n",
-    "\n"
+    "Here we assume each class of image corresponds to a different site."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "id": "e1ea959f-7282-4e55-bb26-11524ec47e99",
    "metadata": {
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Created 4 data lists for ['COVID', 'Lung_Opacity', 'Normal', 'Viral Pneumonia'].\n",
-      "Saved 3616 entries at /tmp/nvflare/image_stats/data/site-1_COVID.json\n",
-      "Saved 6012 entries at /tmp/nvflare/image_stats/data/site-2_Lung_Opacity.json\n",
-      "Saved 10192 entries at /tmp/nvflare/image_stats/data/site-3_Normal.json\n",
-      "Saved 1345 entries at /tmp/nvflare/image_stats/data/site-4_Viral Pneumonia.json\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
-    "from image_stats.utils.prepare_data import prepare_data\n",
+    "from code.image_stats.utils.prepare_data import prepare_data\n",
     "\n",
     "prepare_data(input_dir = \"/tmp/nvflare/image_stats/data\", \n",
     "             input_ext = \".png\",\n",
-    "             output_dir =\"/tmp/nvflare/image_stats/data\")\n",
-    "\n"
+    "             output_dir =\"/tmp/nvflare/image_stats/data\")\n"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "f00de5e4-4360-4fc5-a819-4eb156e56341",
    "metadata": {},
    "source": [
-    "## Run Job in FL Simulator\n",
-    "\n",
+    "## Run Job in FL Simulator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7e972070",
+   "metadata": {},
+   "source": [
+    "The file [image_stats_job.py](code/image_stats_job.py) uses the StatsJob to generate a job configuration in a Pythonic way. With the default arguments, the job will be exported to `/tmp/nvflare/jobs/stats_df` and then the job will be run with the FL simulator with the `simulator_run()` command with a work_dir of `/tmp/nvflare/jobs/stats_df/work_dir`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0db7cd1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! python3 code/image_stats_job.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b3a1214",
+   "metadata": {},
+   "source": [
     "**Run Job with Simulator API**\n",
-    "\n"
+    "\n",
+    "Alternatively, with the job configuration in `code/image_stats/jobs/image_stats`, we can run the job with the Simulator API:"
    ]
   },
   {
@@ -162,7 +163,7 @@
    "outputs": [],
    "source": [
     "from nvflare.private.fed.app.simulator.simulator_runner import SimulatorRunner\n",
-    "runner = SimulatorRunner(job_folder=\"image_stats/jobs/image_stats\", workspace=\"/tmp/nvflare/workspace/image_stats\", n_clients = 4, threads=4)\n",
+    "runner = SimulatorRunner(job_folder=\"code/image_stats/jobs/image_stats\", workspace=\"/tmp/nvflare/workspace/image_stats\", n_clients = 4, threads=4)\n",
     "runner.run()"
    ]
   },
@@ -182,10 +183,7 @@
     "\n",
     "```\n",
     "\n",
-    "assuming the nvflare is installed from a **terminal**. doing pip install from the notebook cell directory with bash command (! or %%bash) may or may not work depending on which python runtime kernel selected. Also %pip install or %pip install from notebook cell doesn't register the console_scripts in the PATH.   \n",
-    "\n",
-    "\n",
-    "## Examine the result\n"
+    "## Examine the result"
    ]
   },
   {
@@ -194,25 +192,17 @@
    "metadata": {},
    "source": [
     "\n",
-    "The results are stored in workspace \"/tmp/nvflare/image_stats\""
+    "The results are stored on the server in the workspace at \"/tmp/nvflare/image_stats\" and can be accessed with the following command:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": null,
    "id": "112a7dd0-45d9-42ea-98b2-f72a3bbccf48",
    "metadata": {
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "-rw-r--r--  1 kevlu  wheel  38015 Jan 30 22:27 /tmp/nvflare/workspace/image_stats/server/simulate_job/statistics/image_statistics.json\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "! ls -al /tmp/nvflare/workspace/image_stats/server/simulate_job/statistics/image_statistics.json"
    ]
@@ -248,47 +238,6 @@
     "now we can visualize via the [visualization notebook](image_stats/demo/visualization.ipynb)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "52d2a52f-8a8d-45ef-8a50-ddbb4ed2f2c6",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "We are not quite done yet. What if you prefer to use python API instead CLI to run jobs. Lets do that in this section"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7e972070",
-   "metadata": {},
-   "source": [
-    "The file [image_stats_job.py](image_stats/job_api/image_stats_job.py) uses the StatsJob to generate a job configuration in a Pythonic way. With the default arguments, the job will be exported to `/tmp/nvflare/jobs/stats_df` and then the job will be run with a work_dir of `/tmp/nvflare/jobs/stats_df/work_dir`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "0db7cd1f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Traceback (most recent call last):\n",
-      "  File \"/Users/kevlu/workspace/repos/NVFlare/examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_image_data/image_stats/job_api/image_stats_job.py\", line 16, in <module>\n",
-      "    from image_statistics import ImageStatistics\n",
-      "  File \"/Users/kevlu/workspace/repos/NVFlare/examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_image_data/image_stats/job_api/image_statistics.py\", line 20, in <module>\n",
-      "    from monai.data import ITKReader, load_decathlon_datalist\n",
-      "ModuleNotFoundError: No module named 'monai'\n"
-     ]
-    }
-   ],
-   "source": [
-    "! python3 image_stats/job_api/image_stats_job.py"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "fda06c0b-798d-480d-9b4c-a62fab95bcf0",

examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_tabular_data/df_stats/job_api/df_stats_job.py → examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_tabular_data/code/df_stats_job.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 import argparse
 
-from df_statistics import DFStatistics
+from src.df_statistics import DFStatistics
 
 from nvflare.job_config.stats_job import StatsJob
 

examples/tutorials/self-paced-training/part-1_federated_learning_introduction/Chapter-2_develop_federated_learning_applications/02.1_federated_statistics/federated_statistics_with_tabular_data/federated_statistics_with_tabular_data.ipynb

@@ -9,11 +9,7 @@
    "source": [
     "# Data Frame Federated Statistics \n",
     "\n",
-    "In this example, we will show how to generate federated statistics for data that can be represented as Pandas Data Frame.\n",
-    "\n",
-    "## Set Up NVFLARE\n",
-    "\n",
-    "Follow [Getting Started](https://nvflare.readthedocs.io/en/main/getting_started.html) to set up a virtual environment and install NVFLARE.\n"
+    "In this example, we will show how to generate federated statistics for data that can be represented as Pandas Data Frame."
    ]
   },
   {
@@ -36,7 +32,7 @@
    },
    "outputs": [],
    "source": [
-    "%pip install -r df_stats/requirements.txt"
+    "%pip install -r code/requirements.txt"
    ]
   },
   {
@@ -49,13 +45,11 @@
     "\n",
     "## Prepare data\n",
     "\n",
-    "In this example, we are using UCI (University of California, Irvine) [adult dataset](https://archive.ics.uci.edu/dataset/2/adult)\n",
-    "The original dataset has already contains \"training\" and \"test\" datasets. Here we simply assume that \"training\" and test data sets are belong to different clients.\n",
-    "so we assigned the training data and test data into two clients.\n",
-    " \n",
-    "Now we use data utility to download UCI datasets to separate client package directory to /tmp/nvflare/data/ directory.\n",
-    "Please note that the UCI's website may experience occasional downtime.\n",
-    "\n"
+    "In this example, we are using the UCI (University of California, Irvine) [adult dataset](https://archive.ics.uci.edu/dataset/2/adult)\n",
+    "The original dataset already contains \"training\" and \"test\" datasets. Here we simply assume that the \"training\" and \"test\" data set each belong to a client, so we assign the adult.train dataset to site-1 and the adult.test dataset to site-2.\n",
+    "\n",
+    "Now we use the data utility to download UCI datasets to separate client package directory to /tmp/nvflare/data/ directory.\n",
+    "Please note that the UCI's website may experience occasional downtime."
    ]
   },
   {
@@ -67,7 +61,7 @@
    },
    "outputs": [],
    "source": [
-    "from df_stats.utils.prepare_data import prepare_data\n",
+    "from code.df_stats.utils.prepare_data import prepare_data\n",
     "\n",
     "prepare_data(data_root_dir = \"/tmp/nvflare/df_stats/data\")"
    ]